/dev/sda1 not a subset of /dev/sda?

the first entry of my partition table is:

$ sudo hexdump -Cv -n 16 -s 446 /dev/sda
000001be  80 01 01 00 83 fe ff ff  3f 00 00 00 81 1c 20 03  |........?..... .|

(-Cv describe the output format, -n 16 asks for 16 bytes and -s 446 skips the first 446 bytes)
You can see that my first partition is a primary Linux partition and that this partition begin at sector 63 (see for example [here][1] for the structure of the partition table).
I would then expect that except for the first 63 sectors and the other partitions, /dev/sda1 and /dev/sda are exactly the same.

But this is not the case, the sector #2 of /dev/sda1 is not exactly the same as the sector #65 of /dev/sda (but they are very similar, only 16 bytes are different):

$ sudo hexdump -Cv -n 512 -s 65b /dev/sda
00008200  00 20 19 00 90 03 64 00  2d 00 05 00 5a 2f 56 00  |. ....d.-...Z/V.|
00008210  b6 b1 16 00 00 00 00 00  02 00 00 00 02 00 00 00  |................|
00008220  00 80 00 00 00 80 00 00  00 20 00 00 d8 38 ee 4c  |......... ...8.L|
00008230  9a 01 ef 4c 05 00 24 00  53 ef 01 00 01 00 00 00  |...L..$.S.......|
00008240  59 23 e9 4c 00 4e ed 00  00 00 00 00 01 00 00 00  |Y#.L.N..........|
00008250  00 00 00 00 0b 00 00 00  00 01 00 00 3c 00 00 00  |............<...|
00008260  42 02 00 00 7b 00 00 00  85 23 eb f2 71 67 44 f5  |B...{....#..qgD.|
00008270  bb 8f 6f f2 3a 59 ff 4d  55 62 75 6e 74 75 00 00  |..o.:Y.MUbuntu..|
00008280  00 00 00 00 00 00 00 00  2f 75 62 75 6e 74 75 00  |......../ubuntu.|
00008290  d8 3c df 5d 00 88 ff ff  52 d0 ef 1d 00 00 00 00  |.<.]....R.......|
000082a0  c0 40 51 b6 00 88 ff ff  00 4e c8 bb 00 88 ff ff  |[email protected]......|
000082b0  c0 f6 86 b8 00 88 ff ff  30 2e 0d a0 ff ff ff ff  |........0.......|
000082c0  38 3d df 5d 00 88 ff ff  00 00 00 00 00 00 fe 03  |8=.]............|
000082d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000082e0  08 00 00 00 00 00 00 00  00 00 00 00 8a 53 d3 0e  |.............S..|
000082f0  7c 7a 43 e4 8b fb ca e0  72 b7 fa c8 01 01 00 00  ||zC.....r.......|
00008300  00 00 00 00 00 00 00 00  16 4c 47 4b 0a f3 03 00  |.........LGK....|
00008310  04 00 00 00 00 00 00 00  00 00 00 00 fe 7f 00 00  |................|
00008320  24 b7 0c 00 fe 7f 00 00  01 00 00 00 22 37 0d 00  |$..........."7..|
00008330  ff 7f 00 00 01 00 00 00  23 37 0d 00 00 00 00 00  |........#7......|
00008340  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 08  |................|
00008350  00 00 00 00 00 00 00 00  00 00 00 00 1c 00 1c 00  |................|
00008360  01 00 00 00 e9 7f 00 00  00 00 00 00 00 00 00 00  |................|
00008370  00 00 00 00 04 00 00 00  9f 7d bb 00 00 00 00 00  |.........}......|
00008380  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00008390  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000083a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000083b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000083c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000083d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000083e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000083f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

versus

$ sudo hexdump -Cv -n 512 -s 2b /dev/sda1
00000400  00 20 19 00 90 03 64 00  2d 00 05 00 5a 2f 56 00  |. ....d.-...Z/V.|
00000410  b6 b1 16 00 00 00 00 00  02 00 00 00 02 00 00 00  |................|
00000420  00 80 00 00 00 80 00 00  00 20 00 00 df 76 ef 4c  |......... ...v.L|
00000430  df 76 ef 4c 06 00 24 00  53 ef 01 00 01 00 00 00  |.v.L..$.S.......|
00000440  59 23 e9 4c 00 4e ed 00  00 00 00 00 01 00 00 00  |Y#.L.N..........|
00000450  0

Solution 1:

Nice finding, as I was able to reproduce this effect on my system, too. At my site it happens on /dev/hda, so it is no SCSI issue.

# uname -a
Linux X.X.X 2.6.26-2-686 #1 SMP Tue Mar 9 17:35:51 UTC 2010 i686 GNU/Linux

I think whitequark is right that it is a cache issue. Here is my interpretation what happened on your site (note that I am not sure that my explanation is correct, though):

/dev/sda1 is in use. So "sync" updates the superblock every time the journal is flushed (or similar). So the disk /dev/sda1 is changed.

However the kernel does not use a combined cache for /dev/sda and /dev/sda1, instead both "files" are cache on it's own. Updating /dev/sda1 (sync) therefor does not invalidate the cache of /dev/sda. Hence reading from /dev/sda shows the old cache value (so the cache is out of sync with the harddrive) while /dev/sda1 shows the correct (new) values.

Here is the situation seen on my side. I came here having done some dumps before on /dev/hda, so it had already cached some old data:

# od -tx1z -N 10k /dev/hda2 > NOW1
# dd ibs=512 skip=1975995 if=/dev/hda | od -tx1z -N 10k > MAIN1
# diff NOW1 MAIN1
3,4c3,4
< 0002000 00 00 31 01 57 c4 61 02 04 7d 1e 00 be 1a 39 00  >..1.W.a..}....9.<
< 0002020 4e ea 21 01 00 00 00 00 02 00 00 00 02 00 00 00  >N.!.............<
---
> 0002000 00 00 31 01 57 c4 61 02 04 7d 1e 00 4e 1b 39 00  >..1.W.a..}..N.9.<
> 0002020 52 ea 21 01 00 00 00 00 02 00 00 00 02 00 00 00  >R.!.............<

# od -tx1z -N 10k /dev/hda2 > NOW1
# diff NOW1 MAIN1
3,4c3,4
< 0002000 00 00 31 01 57 c4 61 02 04 7d 1e 00 be 1a 39 00  >..1.W.a..}....9.<
< 0002020 4e ea 21 01 00 00 00 00 02 00 00 00 02 00 00 00  >N.!.............<
---
> 0002000 00 00 31 01 57 c4 61 02 04 7d 1e 00 4e 1b 39 00  >..1.W.a..}..N.9.<
> 0002020 52 ea 21 01 00 00 00 00 02 00 00 00 02 00 00 00  >R.!.............<

# od -tx1z -N 10k /dev/hda2 > NOW2
# dd ibs=512 skip=1975995 if=/dev/hda | od -tx1z -N 10k > MAIN2
# diff MAIN1 MAIN2
# diff NOW2 MAIN2
3,4c3,4
< 0002000 00 00 31 01 57 c4 61 02 04 7d 1e 00 f0 19 39 00  >..1.W.a..}....9.<
< 0002020 41 ea 21 01 00 00 00 00 02 00 00 00 02 00 00 00  >A.!.............<
---
> 0002000 00 00 31 01 57 c4 61 02 04 7d 1e 00 4e 1b 39 00  >..1.W.a..}..N.9.<
> 0002020 52 ea 21 01 00 00 00 00 02 00 00 00 02 00 00 00  >R.!.............<
106c106
< 0012440 00 80 14 00 01 80 14 00 02 80 14 00 00 00 01 00  >................<
---
> 0012440 00 80 14 00 01 80 14 00 02 80 14 00 00 00 00 00  >................<
334c334
< 0021540 00 80 4d 00 01 80 4d 00 02 80 4d 00 02 00 63 3e  >..M...M...M...c><
---
> 0021540 00 80 4d 00 01 80 4d 00 02 80 4d 00 02 00 64 3e  >..M...M...M...d><

While /dev/hda does not show any update, /dev/hda2 shows some changes. But when I flush the caches and try again, everything shows to be the same:

# hdparm -f /dev/hda

/dev/hda:

# dd ibs=512 skip=1975995 if=/dev/hda | od -tx1z -N 10k > MAIN2
# diff MAIN1 MAIN2
3,4c3,4
< 0002000 00 00 31 01 57 c4 61 02 04 7d 1e 00 4e 1b 39 00  >..1.W.a..}..N.9.<
< 0002020 52 ea 21 01 00 00 00 00 02 00 00 00 02 00 00 00  >R.!.............<
---
> 0002000 00 00 31 01 57 c4 61 02 04 7d 1e 00 dc 1a 39 00  >..1.W.a..}....9.<
> 0002020 96 ea 21 01 00 00 00 00 02 00 00 00 02 00 00 00  >..!.............<
36,37c36,37
< 0010300 00 00 03 00 01 00 03 00 02 00 03 00 00 00 bb 3b  >...............;<
< 0010320 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >................<
---
> 0010300 00 00 03 00 01 00 03 00 02 00 03 00 00 00 bc 3b  >...............;<
> 0010320 7f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >................<
48c48
< 0010600 00 00 06 00 01 00 06 00 02 00 06 00 03 00 18 3f  >...............?<
---
> 0010600 00 00 06 00 01 00 06 00 02 00 06 00 04 00 18 3f  >...............?<
106c106
< 0012440 00 80 14 00 01 80 14 00 02 80 14 00 00 00 00 00  >................<
---
> 0012440 00 80 14 00 01 80 14 00 02 80 14 00 00 00 01 00  >................<

# od -tx1z -N 10k /dev/hda2 > NOW2
# diff NOW2 MAIN2
# diff MAIN1 MAIN2
3,4c3,4
< 0002000 00 00 31 01 57 c4 61 02 04 7d 1e 00 4e 1b 39 00  >..1.W.a..}..N.9.<
< 0002020 52 ea 21 01 00 00 00 00 02 00 00 00 02 00 00 00  >R.!.............<
---
> 0002000 00 00 31 01 57 c4 61 02 04 7d 1e 00 dc 1a 39 00  >..1.W.a..}....9.<
> 0002020 96 ea 21 01 00 00 00 00 02 00 00 00 02 00 00 00  >..!.............<
36,37c36,37
< 0010300 00 00 03 00 01 00 03 00 02 00 03 00 00 00 bb 3b  >...............;<
< 0010320 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >................<
---
> 0010300 00 00 03 00 01 00 03 00 02 00 03 00 00 00 bc 3b  >...............;<
> 0010320 7f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >................<
48c48
< 0010600 00 00 06 00 01 00 06 00 02 00 06 00 03 00 18 3f  >...............?<
---
> 0010600 00 00 06 00 01 00 06 00 02 00 06 00 04 00 18 3f  >...............?<
106c106
< 0012440 00 80 14 00 01 80 14 00 02 80 14 00 00 00 00 00  >................<
---
> 0012440 00 80 14 00 01 80 14 00 02 80 14 00 00 00 01 00  >................<

Short note on how to reproduce:

  • Make sure your system is idle and has plenty of RAM for caching.
  • run fdisk -u -l to find where the partition starts. At my side it is 1975995
  • Choose a mounted partition and the complete drive.
  • Then do the both dumps (NOW1, MAIN1) and compare them. They shall compare equal.
  • Change a bit on the partition, sync.
  • Leave it more time.
  • Change a bit, sync again.
  • Do dump NOW2, it shall differ from NOW1
  • Do a dump MAIN2, it should not differ from MAIN1!
  • Execute the hdparm line for your drive
  • Do dump MAIN2 again, now it will differ from MAIN1.
  • If you are quick enough, NOW2 and MAIN2 will compare equal.