Housekeeper | 2023-05-26 10:43:06 | ERROR | Housekeeper | start structure: [source /entrypoint.sh && structureStartup] | /entrypoint.sh: line 842: 71263 Killed /usr/local/dotnet-3.1.0-aspnetcore-runtime/dotnet {structureDir}/MD.Structure.GRPCService.dll | exit status 137
Housekeeper | 2023-05-26 10:43:28 | INFO | Housekeeper | stop structure: [source /entrypoint.sh && structureShutdown]
Housekeeper | 2023-05-26 10:43:30 | INFO | Housekeeper | start structure: [source /entrypoint.sh && structureStartup]
Housekeeper | 2023-05-26 10:53:06 | ERROR | Housekeeper | start structure: [source /entrypoint.sh && structureStartup] | /entrypoint.sh: line 842: 82726 Killed /usr/local/dotnet-3.1.0-aspnetcore-runtime/dotnet {structureDir}/MD.Structure.GRPCService.dll | exit status 137
Housekeeper | 2023-05-26 10:53:28 | INFO | Housekeeper | stop structure: [source /entrypoint.sh && structureShutdown]
Housekeeper | 2023-05-26 10:53:30 | INFO | Housekeeper | start structure: [source /entrypoint.sh && structureStartup]
Housekeeper | 2023-05-26 11:03:03 | ERROR | Housekeeper | start structure: [source /entrypoint.sh && structureStartup] | /entrypoint.sh: line 842: 92849 Killed /usr/local/dotnet-3.1.0-aspnetcore-runtime/dotnet ${structureDir}/MD.Structure.GRPCService.dll | exit status 137
Housekeeper | 2023-05-26 11:03:28 | INFO | Housekeeper | stop structure: [source /entrypoint.sh && structureShutdown]
Housekeeper | 2023-05-26 11:03:30 | INFO | Housekeeper | start structure: [source /entrypoint.sh && structureStartup]
👀️ 观测云好用吗
Alias 2023-05-30 09:17:55每天零晨 1 点服务器上在做备份,所以 CPU 占用比较高,
内存的使用自升级到 3.6 后就不太正常,26 号 12 点钟,降到 3.5.3 后内存占用回到正常
看监控曲线,内存监控有达到 100% 的情况,这个时候可能导致服务异常了。
Alias 2023-05-30 09:17:55每天零晨 1 点服务器上在做备份,所以 CPU 占用比较高,
内存的使用自升级到 3.6 后就不太正常,26 号 12 点钟,降到 3.5.3 后内存占用回到正常
不知道这个现像正常吗,每秒钟很多个这样的日志
Alias 2023-05-30 09:14:31
每天零晨 1 点服务器上在做备份,所以 CPU 占用比较高,
内存的使用自升级到 3.6 后就不太正常,26 号 12 点钟,降到 3.5.3 后内存占用回到正常
禹建勋(Team) 2023-05-29 15:05:49有服务器最近一周的内存和 CPU 的监控曲线图吗?
禹建勋(Team) 2023-05-29 12:03:01服务器上是只有明道服务吗?看报错是资源不足杀掉了一些进程
退回到 3.5.3 后,应用日志再无报错
Alias 2023-05-29 15:04:37这台服务器只专用于跑明道云的,而配置也是很高
62G 内存,32 核 CPU。
我们每天的工作流大概也是和以下图差不多,
有服务器最近一周的内存和 CPU 的监控曲线图吗?
禹建勋(Team) 2023-05-29 12:03:01服务器上是只有明道服务吗?看报错是资源不足杀掉了一些进程
这台服务器只专用于跑明道云的,而配置也是很高
62G 内存,32 核 CPU。
我们每天的工作流大概也是和以下图差不多,
Alias 2023-05-29 11:05:11MongoDB 没添加认证的,这个错误报的很少
服务器上是只有明道服务吗?看报错是资源不足杀掉了一些进程
禹建勋(Team) 2023-05-29 10:32:16MongoDB 是不是添加了认证
MongoDB 没添加认证的,这个错误报的很少
Alias 2023-05-29 10:29:18当时出问题时,看了内存使用情况,还算好,当时没截图保存。
以下图片是现在 3.5.3 版本下的截图:
重启过几次微服务。版本从 3.6.0 开始就出现这现像,升级到 3.7.2 后问题同样。现在回退 3.5.3 版本再没有此现像。
3.7.2 报错信息有以下几种:
一、节点: 新增审批记录, org.springframework.dao.DataIntegrityViolationException: Cannot create index for '' in collection 'code_config' with keys 'Document{{p=1, n=1}}' and options 'Document{{name=code_config_idx, unique=true, background=true}}'.; nested exception is com.mongodb.MongoCommandException: Command failed with error 85 (IndexOptionsConflict): 'Index: { v: 2, unique: true, key: { p: 1, n: 1 }, name: "code_config_idx", ns: "mdworkflow.code_config", background: true } already exists with different options: { v: 2, key: { p: 1.0, n: 1.0 }, name: "p_1_n_1", ns: "mdworkflow.code_config" }' on server 127.0.0.1:27017. The full response is {"ok": 0.0, "errmsg": "Index: { v: 2, unique: true, key: { p: 1, n: 1 }, name: "code_config_idx", ns: "mdworkflow.code_config", background: true } already exists with different options: { v: 2, key: { p: 1.0, n: 1.0 }, name: "p_1_n_1", ns: "mdworkflow.code_config" }", "code": 85, "codeName": "IndexOptionsConflict"}
二、应用运行情况服务不稳定
Housekeeper | 2023-05-26 10:43:06 | ERROR | Housekeeper | start structure: [source /entrypoint.sh && structureStartup] | /entrypoint.sh: line 842: 71263 Killed /usr/local/dotnet-3.1.0-aspnetcore-runtime/dotnet {structureDir}/MD.Structure.GRPCService.dll | exit status 137 Housekeeper | 2023-05-26 10:43:28 | INFO | Housekeeper | stop structure: [source /entrypoint.sh && structureShutdown] Housekeeper | 2023-05-26 10:43:30 | INFO | Housekeeper | start structure: [source /entrypoint.sh && structureStartup] Housekeeper | 2023-05-26 10:53:06 | ERROR | Housekeeper | start structure: [source /entrypoint.sh && structureStartup] | /entrypoint.sh: line 842: 82726 Killed /usr/local/dotnet-3.1.0-aspnetcore-runtime/dotnet {structureDir}/MD.Structure.GRPCService.dll | exit status 137
Housekeeper | 2023-05-26 10:53:28 | INFO | Housekeeper | stop structure: [source /entrypoint.sh && structureShutdown]
Housekeeper | 2023-05-26 10:53:30 | INFO | Housekeeper | start structure: [source /entrypoint.sh && structureStartup]
Housekeeper | 2023-05-26 11:03:03 | ERROR | Housekeeper | start structure: [source /entrypoint.sh && structureStartup] | /entrypoint.sh: line 842: 92849 Killed /usr/local/dotnet-3.1.0-aspnetcore-runtime/dotnet ${structureDir}/MD.Structure.GRPCService.dll | exit status 137
Housekeeper | 2023-05-26 11:03:28 | INFO | Housekeeper | stop structure: [source /entrypoint.sh && structureShutdown]
Housekeeper | 2023-05-26 11:03:30 | INFO | Housekeeper | start structure: [source /entrypoint.sh && structureStartup]
MongoDB 是不是添加了认证
禹建勋(Team) 2023-05-29 09:33:20服务器总资源多大?free -g 看下。
重启微服务是否可以正常?
当时出问题时,看了内存使用情况,还算好,当时没截图保存。
以下图片是现在 3.5.3 版本下的截图:
重启过几次微服务。版本从 3.6.0 开始就出现这现像,升级到 3.7.2 后问题同样。现在回退 3.5.3 版本再没有此现像。
3.7.2 报错信息有以下几种:
一、节点: 新增审批记录, org.springframework.dao.DataIntegrityViolationException: Cannot create index for '' in collection 'code_config' with keys 'Document{{p=1, n=1}}' and options 'Document{{name=code_config_idx, unique=true, background=true}}'.; nested exception is com.mongodb.MongoCommandException: Command failed with error 85 (IndexOptionsConflict): 'Index: { v: 2, unique: true, key: { p: 1, n: 1 }, name: "code_config_idx", ns: "mdworkflow.code_config", background: true } already exists with different options: { v: 2, key: { p: 1.0, n: 1.0 }, name: "p_1_n_1", ns: "mdworkflow.code_config" }' on server 127.0.0.1:27017. The full response is {"ok": 0.0, "errmsg": "Index: { v: 2, unique: true, key: { p: 1, n: 1 }, name: "code_config_idx", ns: "mdworkflow.code_config", background: true } already exists with different options: { v: 2, key: { p: 1.0, n: 1.0 }, name: "p_1_n_1", ns: "mdworkflow.code_config" }", "code": 85, "codeName": "IndexOptionsConflict"}
二、应用运行情况服务不稳定
Housekeeper | 2023-05-26 10:43:06 | ERROR | Housekeeper | start structure: [source /entrypoint.sh && structureStartup] | /entrypoint.sh: line 842: 71263 Killed /usr/local/dotnet-3.1.0-aspnetcore-runtime/dotnet {structureDir}/MD.Structure.GRPCService.dll | exit status 137 Housekeeper | 2023-05-26 10:43:28 | INFO | Housekeeper | stop structure: [source /entrypoint.sh && structureShutdown] Housekeeper | 2023-05-26 10:43:30 | INFO | Housekeeper | start structure: [source /entrypoint.sh && structureStartup] Housekeeper | 2023-05-26 10:53:06 | ERROR | Housekeeper | start structure: [source /entrypoint.sh && structureStartup] | /entrypoint.sh: line 842: 82726 Killed /usr/local/dotnet-3.1.0-aspnetcore-runtime/dotnet {structureDir}/MD.Structure.GRPCService.dll | exit status 137
Housekeeper | 2023-05-26 10:53:28 | INFO | Housekeeper | stop structure: [source /entrypoint.sh && structureShutdown]
Housekeeper | 2023-05-26 10:53:30 | INFO | Housekeeper | start structure: [source /entrypoint.sh && structureStartup]
Housekeeper | 2023-05-26 11:03:03 | ERROR | Housekeeper | start structure: [source /entrypoint.sh && structureStartup] | /entrypoint.sh: line 842: 92849 Killed /usr/local/dotnet-3.1.0-aspnetcore-runtime/dotnet ${structureDir}/MD.Structure.GRPCService.dll | exit status 137
Housekeeper | 2023-05-26 11:03:28 | INFO | Housekeeper | stop structure: [source /entrypoint.sh && structureShutdown]
Housekeeper | 2023-05-26 11:03:30 | INFO | Housekeeper | start structure: [source /entrypoint.sh && structureStartup]
Alias 2023-05-29 09:16:41
不好意思,当时出错的时候只有这个截图,不知道能不能帮到你分析
服务器总资源多大?free -g 看下。
重启微服务是否可以正常?
张春泉(Team) 2023-05-26 17:52:34服务器资源使用率看下。
不好意思,当时出错的时候只有这个截图,不知道能不能帮到你分析
服务器资源使用率看下。