Crash: systemd-journal Failed to write entry. Ignoring: read-only file system only

Solution 1:

The short answer: as suggested here on the Arch wiki, try updating the firmware of your SSD.

Now the WHY, (only for those who would like to understand, how did I get to this conclusion, perhaps learning some useful tips in this type of troubleshooting - or better, to politely correct me as needed):

I have the same problem, and I have Ubuntu 18.04 too. I have my root partition mounted on a Samsung SSD 860 EVO 500GB. You seem to have an ADATA SSD.

I have seen many posts with this question, no solution, and usual answers warn about possible trouble with the SSD filesystem, corrupted/damage device, and so on. I don't think so.

For SSD, Ubuntu enables periodic TRIM by default. That happens via fstrim.service, on a weekly basis.

According to man fstrim:

Running fstrim frequently, or even using mount -o discard, 
might negatively affect the lifetime of poor-quality SSD devices. 
For most desktop and server systems a sufficient trimming frequency is once a week. 
Note that not all devices support a queued trim, so each trim command 
incurs a performance penalty on whatever else might be trying to use the disk at the time.

...which suggests me the following scenario and my own hypothesis (which I support below with my own research):

When fstrim is running, (weekly, in this case) it certainly needs to lock the filesystem it's trimming. Meanwhile, journald is trying to write some logs in /var/log/journal/, and it finds it can't write it (according to me, because fstrim is busy with / or with the SSD, in any case). Thus journald's complaint: Failed to write entry (...) Read-only file system.

"But: wait! That doesn't make sense" you may interject at this point. Well, indeed, those services should not be fighting for writing the filesystem or locking it at the same time. But the most compelling clue comes from the above man pages citation: Note that not all devices support a queued trim, so each trim command incurs a performance penalty on whatever else might be trying to use the disk at the time.

...which is consistent with what we see happening when we boot on the day our fstrim.service decide to do its thing in a very rude way, in total disregard for the expected "etiquette" that asynchronous services should follow.

In fact, if you look at journalctl -u fstrim.service, as I did, you might see a sequence of many blocks similar to this:

-- Reboot --
Jul 13 00:00:13 foo systemd[1]: Starting Discard unused blocks...
Jul 13 00:02:43 foo fstrim[19452]: /Some/MountPoint: 79,4 GiB (85273759744 bytes) trimmed
Jul 13 00:02:43 foo fstrim[19452]: /: 22,4 GiB (24017248256 bytes) trimmed
Jul 13 00:02:43 foo systemd[1]: Started Discard unused blocks.

which is what I have from one week ago, when I had, again, the aforementioned problem. And yet again, today, the same thing. I had to reboot.

Note that the time between "Starting" and "Started" is between around 150 seconds (2m30s). In my logs, this is always very similar; sometimes perhaps about 3 to 3.5 minutes, no more. And nothing there indicates (in my case) that the trimming failed. Either the service doesn't log success or failure (I don't know) or this messages tell "Done".

It still may sound strange, in my hypothesis, that if the messages show an interval of a couple of hundred of seconds, why that would disturb journald writing for a huge amount of time? (oh, it does: I have waited a lot of time in those cases and errors won't stop coming!). Well, I can only guess that the errors come only after several retries and/or the system got so many errors queued to print to the terminal, and somehow this gets very slow to get done. This is only my guess: If anyone could explain this in a technically accurate way, please, by all means, do help or do correct me.

But it is a hard fact, that after this "almost hanging" on boot, the journal gets corrupted. In my case, I got this in the output of dmesg, just the next successful boot after the "semi-hang":

[   10.986870] systemd-journald[434]: File /var/log/journal/414716237ed94301a0e84b4204143c5b/system.journal corrupted or uncleanly shut down, renaming and replacing.

It makes sense, because I did a brute reboot.

The final take and the point of my hypothesis, is that my (and probably yours too) SSD device is lacking support for queued trim. The good news, in my hypothesis, is that that means we are not having corrupted SSD filesystem or hardware. It means that we are getting troubles with the system.journal file that system-journald service is trying to create.

And, short answer: above all, we should try and update the firmware of our SSD.

Solution 2:

Try upgrading your kernel.

I was getting this same "write entry" crash every night after a recent update on Ubuntu 20.04.

I am using a cheap SSD without DRAM and without an option to update the firmware like the other answer suggests. Eventually I read somewhere that the system can lock and incorrectly mark areas as "read-only" if the kernel isn't working correctly, so I updated to a newer kernel (5.7.1) and haven't had the issue since.

Here's an article on how to upgrade the kernel

If you suspect your problem is the kernel and want to test this theory first, you can go into "Advanced Options" on the boot screen after a restart and select an older version of the kernel to run with. If that kernel works for you then you might want to set that as the kernel to use for now instead, as the newest kernels are unsigned and either require signing them yourself or turning off secure boot.