mesos task Failed to create the cgroup at...
完整的错误信息:
Container ‘bf7a0add-5d1e-4484-a438-860ed2e72be9’ for executor ‘ct:1517751223061:1:69_dir_users_fix:’ of framework 0ee72276-2777-4d36-a82b-7e46f055f69b-0000 failed to start: Failed to create the cgroup at ‘/sys/fs/cgroup/devices/mesos/bf7a0add-5d1e-4484-a438-860ed2e72be9’: Failed to create directory ‘/sys/fs/cgroup/devices/mesos/bf7a0add-5d1e-4484-a438-860ed2e72be9’: No such file or directory
查日志/var/log/message发现,某个时间点突然/sys/fs/cgroup/devices目录被删了
搜索最终在mesos jira上发现如下issue
Failed to create the cgroup at …
/sys/fs/cgroup/memory/mesos missing after running a while
第一个和我遇到的问题一样但没有解决办法,第二个比较类似,评论里有解决方法,
也是某个cgroup下的某个目录丢了,但不是mesos的bug,是systemd会不定期清理cgroup资源,算是systemd的小bug,因为systemd默认会把用户创建的cgroup也清理掉。
bug详情
/sys/fs/cgroup/cpu,cpuacct/machine.slice disappears after systemctl daemon-reload
Should not destroy application created cgroups
解决办法就是把systemd升级从208(centos7.1默认)升级到219(centos7.2默认),添加参数
Delegate=true
当然还有更暴力的方法,就是添加脚本,检查这个目录如果不存在就创建
#!/bin/bash
dir="/sys/fs/cgroup/devices/mesos"
date
if [ ! -d $dir ]; then
echo "mkdir now..."
mkdir -p $dir
fi
Comments