merve04

Errors in file system?

Recommended Posts

I'm not sure if this is where I need to post this but here we go,

Yesterday i was tinkering around in docker downloading images, removing old images, doing some backups etc all very simultaneously. Then a box popped up mentioning errors occurred in the file systems....

I've tried doing a simple reboot of the machine but issues still there, and i tried doing the "reboot now" and run a check, maybe 30min went by, maybe less, just recall seeing "checking quota" and then i was able to log in dsm again. Heres some screenshots.

 

Screen Shot 2020-09-06 at 7.06.54 AM.png

Screen Shot 2020-09-06 at 7.07.07 AM.png

Screen Shot 2020-09-06 at 7.07.15 AM.png

Screen Shot 2020-09-06 at 7.07.30 AM.png

Share this post


Link to post
Share on other sites

Is there a question that needs an answer here?

 

It looks like you must be using ext4 filesystem instead of btrfs and the system decided it had some corruption and recommended an automatic fsck.  It can only do that when the ext4 system is unmounted, so that means on boot.  Not unusual for it to take 30 minutes or longer, I have some very large blob ext4 filesystems at work that take over 24 hours to fsck.

 

If it were btrfs you would get a similar notification of the error but it would have been fixed on the spot if it was possible via redundancy.

Share this post


Link to post
Share on other sites

Indeed there’s a question, twice I’ve tried reboot now and run, both times come back to dsm stating there’s an error. Is there anything else i can do?

Share this post


Link to post
Share on other sites

The message seems to be saying that DSM cannot fix the filesystem and that you should offload your files, (delete the filesystem), and recreate it.

Share this post


Link to post
Share on other sites

I’ve done a bit of searching on fsck, I guess there are more complex options for scanning the volume?!? Not too sure if that’s a route I should take. My biggest concern is if I indeed need to recreate the volume, how does one approach offloading nearly 40TB?!?

Share this post


Link to post
Share on other sites

How do you backup your data now?

I agree that's a lot of data.  When it starts to get that voluminous, your backup/restore strategy needs to be adjusted to remain viable and realistic.

 

For me, I keep another "small" XPE box online with very high density drives and its only job is to receive btrfs snapshot replicas.

Share this post


Link to post
Share on other sites

Being that a 8TB drive costs $200, even if i wanted to be cheap\risky\crazy and degrade my volume to use couple of the redundant drives as means of storage, id be set back $600. I've been looking at cloud options, g suite as an example.

I'm also thinking of trying and removing all docker packages and docker app itself and see if that may be the root cause of this error as it did occur while manipulating things in there.

Edited by merve04

Share this post


Link to post
Share on other sites

I'm running data scrub on storage pool 1, could it possibly fix the error or am i wasting my time?

Share this post


Link to post
Share on other sites

Well I tried my second plan as this all started when I was in docker, I removed docker and selected the check mark to remove all settings, files and folder, while it was removing the message about file system error occurred popped up. When it finished removing i did a run/reboot, this time it took almost 1 hour checking quota, when i logged back in dsm, voila warning was gone. 😎

Share this post


Link to post
Share on other sites

Can you confirm your filesystem is on ext4?  AFAIK Docker is not supported unless you are running on btrfs.  Could be part of your problem.

 

# cat /etc/fstab

Share this post


Link to post
Share on other sites

I can confirm my volume is ext4, I can also ensure that I’ve been using docker for probably 2 years now. I do have a second volume as you can see in the picture above for surveillance and the only reason it’s btrfs was to support virtual machine manager. My system is back in checking quota but sure I could install docker on that volume. I’m just lead to suspect something is still lingering after uninstalling docker from volume1

Share this post


Link to post
Share on other sites

so back in dsm now, error gone, decided to ssh in server, theres still a "@docker" folder with some stuff in it, could this be the cause? safe to delete?

Screen Shot 2020-09-09 at 6.38.58 PM.png

Share this post


Link to post
Share on other sites

so interestingly as i kept digging deeper and deeper in the folder structure of @docker, i had dsm logged in the background and on that last CD command, the system error occurred popped up.

Screen Shot 2020-09-09 at 6.50.17 PM.png

Share this post


Link to post
Share on other sites

Anyone know how I can delete the directory? When I do a rm -rf it’s gives me an error that structure needs cleaning. No matter how many times fsck is ran, the problem remains. 

Share this post


Link to post
Share on other sites

So im trying to see if debugfs can remove this folder, but i dont understand why when i try to mount the volume in questions it seems like it just does nothing, i try a "ls" after what I believe should of mounted the volume in debug but nothing happens. Am I missing something, doing something wrong?

Screen Shot 2020-09-12 at 5.38.28 PM.png

Share this post


Link to post
Share on other sites

I don't know much about debugfs but it's really for kernel debugging and passing kernel parameters to userspace.  And I am pretty sure you can't use it on a mounted ext4 filesystem.

 

I know you are trying not to offload your files and rebuild with the hope of a quick easy fix. I don't mean to be preachy, but trying to edit your filesystem directly is a path that increasingly puts your data at risk. If fsck can't fix it, there is something significantly wrong.

Share this post


Link to post
Share on other sites

Do you or anyone else seeing this know how to perform e2fsck? Ive also read stuff about it being more effective.

10 hours ago, flyride said:

I know you are trying not to offload your files and rebuild with the hope of a quick easy fix.

The reason I'm investing my time in this is i can literally access every other folder in my volume no issue, but as soon as i touch @docker, immediately triggers dsm a error and to run a fs.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.