Memory access violation
COBOL error at 000000 in [[program]]
strace
I found that the runtime interpreter was segfaulting immediately after loading the program, before even trying to run any COBOL code at all.noexec=off
smb.conf
:vfs objects = full_audit
full_audit:priority = INFO
full_audit:facility = LOCAL1
full_audit:failure = none
full_audit:success = mkdir rename unlink rmdir pwrite
full_audit:prefix = %u|%U|%I|%m|%S
/etc/syslog.conf
accordingly, and you have an audit trail.The change log is:
--average-rate
(Henry Gebhardt)--help
(Sebastian Kayser)pv_display
-O
" for non-GCC (Kjetil Torgrim Homme)DESTDIR
/ suffix (Sam Nelson, Daniel Pape)
The main user-visible changes are the new -a
/ --average-rate
option from Henry Gebhardt which gives a much more sensible display for "bursty" traffic, and a consistent, documented exit status so it's easier for scripts to tell when there has been an error.
top
and mutt
within it to see how it
handled. As a result I've fixed a few bugs in escape sequence handling
and line wrapping as well as adding TAB stop support and callbacks for
title changes and other private message strings.script
logs or screen
history into a flat file you can read with less
without all the cursor positioning stuff getting in the way. ]]>
scp
, and ntpstat
commands. Those servers that weren't brand new had not exhibited that
behaviour with RHEL 4 in the past, it was only when Red Hat Enterprise
Linux 5 was installed that it began.ntpstat
would fail to run about 10 times in every 50000, or 0.02% of the time. Each failure, according to strace
, was not actually with the program itself but with the attempt to run it - the execve
call, which causes the program to be executed, was failing with an EINVAL
error code, indicating some sort of problem to do with the ELF interpreter.prelink
./etc/sysconfig/prelink
and by running prelink -au
. Immediately after doing that, ntpstat
worked 100% of the time instead of 99.98%.prelink
's
address space randomisation was breaking stuff on the servers I'm
using, but I am not in a position to test that or to try to find a
proper fix, so for now it remains disabled.prelink -au...and see if your problems disappear.
sed -i s/^PRELINKING=yes/PRELINKING=no/ /etc/sysconfig/prelink
prelink
:Options to prelink | Test results | Success |
---|---|---|
-au | 50000/50000 | 100.00% |
-amR | 49988/50000 | 99.98% |
-aR | 49986/50000 | 99.97% |
-am | 50000/50000 | 100.00% |
prelink -au
run after it followed by another test to make sure success went back to 100%.-R
option to prelink
seems to be the one that's cocking everything up.--rate-limit
(-L
) option, and two new features.--line-mode
(-l
) was a Debian wishlist request.
This causes PV to count lines instead of bytes. While it's not
something I have ever particularly wanted myself, it does sound like it
might come in handy occasionally (and, more importantly, it didn't
require much to be added to the code to make it work).--remote
(-R
)
option allows the settings of an already-running PV to be altered. This
can be used to change the rate limit while a transfer is in progress,
for example, or set PV's idea of the total size of all data to something
different. ]]>
awk '{print $NF}' /proc/*/maps \
| sort - <(for A in /proc/*/exe; do readlink $A; done) \
| uniq \
| grep / \
| while read FILE; do rpm -q --queryformat='%{NAME}\n' -f $FILE 2>/dev/null; done \
| grep -v 'is not owned' \
| sort \
| uniq
--queryformat='%{NAME}\n'
part if you want the RPM version numbers to be included, in case you have multiple versions of some packages installed.readlink
, you could try ls -l $A | awk '{print $NF}'
instead.hdparm -t long /dev/hda
)
showed that it was failing, so I requested that the server operators
replace the hard disk (it's a leased server not under my direct
control)./backup
directory on the new disk, which they had installed a fresh copy of Fedora Core 4 on to./backup/
with the current /
, i.e.
replace the whole new root filesystem with the old one to get
everything back as it was before the change. I also had to do this live
over the network, since I have no physical access to the server.mv /bin /OLD/bin; mv /backup/bin /bin
" you will run into major problems - after the first mv
, the mv
command won't work because it has moved to a different directory, and
if you work around that, you will still eventually run into nasty
problems related to /lib
.mount --bind / /backup/mnt
to make the root filesystem visible under /backup
so that I could then do chroot /backup
(making sure /backup/dev/
was populated with device files first). I was then able to replace, under /backup
, /mnt/{bin,boot,etc,lib,...}
to replace the "real" root filesystem's files (i.e. mv /mnt/bin /mnt/old-bin; cp -a /bin /mnt/bin
etc)./dev
, /proc
, /sys
, or any other directories that are mount points or which have mount points under them (though you could investigate mount --move
). ]]>
# telinit 2
# for SERVICE in \
`chkconfig --list | grep 2:on | awk '{print $1}' | grep -v -e sshd -e network -e rawdevices`; \
do service $SERVICE stop; done
# service nfs stop
# service rpcidmapd stop
# setenforce 0
# umount -a
# mkdir /tmp/tmprootNote that this used up about 1.6GB of ramdisk on my Red Hat Enterprise Linux (AS) 4 server.
# mount none /tmp/tmproot -t tmpfs
# mkdir /tmp/tmproot/{proc,sys,usr,var,oldroot}
# cp -ax /{bin,etc,mnt,sbin,lib} /tmp/tmproot/
# cp -ax /usr/{bin,sbin,lib} /tmp/tmproot/usr/
# cp -ax /var/{account,empty,lib,local,lock,nis,opt,preserve,run,spool,tmp,yp} /tmp/tmproot/var/
# cp -a /dev /tmp/tmproot/dev
/lib64
and /usr/lib64
as well, otherwise you will see errors like "lib64/ld-linux-x86-64.so.2: bad ELF interpreter: No such file or directory".# pivot_root /tmp/tmproot/ /tmp/tmproot/oldroot
# mount none /proc -t proc
# mount none /sys -t sysfs (this may fail on 2.4 systems)
# mount none /dev/pts -t devpts
# service sshd restartYou should now try to make a new connection. If that succeeds, close your old one to release the old pty device. If it fails, get the SSH daemon properly restarted before proceeding.
# umount /oldroot/procNow try to find other things that are still holding on to the old filesystem, particularly
# umount /oldroot/dev/pts
# umount /oldroot/selinux
# umount /oldroot/sys
# umount /oldroot/var/lib/nfs/rpc_pipefs
/dev
:# fuser -vm /oldroot/devCommon processes that will need killing:
# killall udevdFinally, you will need to re-execute
# killall gconfd-2
# killall mingetty
# killall minilogd
init
:# telinit u
# umount -l /oldroot/devNote that we use the
# umount /oldroot
umount -l
("lazy") option, available only with kernels 2.4.11 and later, because /oldroot
is actually mounted using an entry in /oldroot/dev
, so it would be difficult if not impossible to unmount either of them otherwise.# e2fsck -C 0 -f /dev/VolGroup00/LogVol00In this example the root partition is
# resize2fs -p -f /dev/VolGroup00/LogVol00 8G
# lvresize /dev/VolGroup00/LogVol00 -L 8G
# resize2fs -p -f /dev/VolGroup00/LogVol00
# e2fsck -C 0 -f /dev/VolGroup00/LogVol00
/dev/VolGroup00/LogVol00
and it is being shrunk to 8GB. You don't necessarily have to run resize2fs
twice, I just do in case my idea of the size differs from what lvresize
thinks.# mount /dev/VolGroup00/LogVol00 /oldrootNow make a new SSH connection, and if it works, close the old one. Note that
# pivot_root /oldroot /oldroot/tmp/tmproot
# umount /tmp/tmproot/proc
# mount none /proc -t proc
# cp -ax /tmp/tmproot/dev/* /dev/
# mount /dev/pts
# mount /sys
# killall mingetty
# telinit u
# service sshd restart
sshd
may still be running in the temporary filesystem at this point because of the way the service
scripts work - check this with fuser
, and if this is the case, kill the oldest sshd
process and then do service sshd start
. Then log in again and disconnect all other connections.# umount -l /tmp/tmproot/dev/ptsNow to re-mount our original filesystems and start services back up:
# umount -l /tmp/tmproot
# rmdir /tmp/tmproot
# mount -aReplace
# umount /sys
# mount /sys
# for SERVICE in \
`chkconfig --list | grep 2:on | awk '{print $1}' | grep -v -e sshd -e network -e rawdevices`; \
do service $SERVICE start; done
# telinit 3
3
with your preferred runlevel. You may also want to start SELinux up again with setenforce
.pivot_root
, tmpfs
, and umount -l
, so long as you can replace the chkconfig
and service
parts with whatever is appropriate for your distribution.sshd
in step 5 until I did this: mount none /dev/pts -t devpts
"./lib64
and /usr/lib64
, and that after pivot_root
2.6 kernels will also need mount none /sys -t sysfs
and mount none /dev/pts -t devpts
. (The above steps have been modified accordingly). ]]>
I started off with something like what this blog post describes, and it all seemed to be running fine, so I left it for a while.
When I came back, it seems that the machine just got bored and stopped part-way through the installation of the packages. It had installed about half of them, and not done any of the "cleanup" phase yet, when it just aborted (with no error message) and dropped back to the shell. I'm guessing it ran out of memory (that machine has no swap, though it has 512MB of RAM), but there were no logs to confirm that.
So, I switched on swap before continuing.
I tried a second yum update
,
but it failed with conflicts and missing dependencies. This turned out
to be because lots of Fedora Core 6 packages had been installed, but the
corresponding Fedora Core 5 packages had not been uninstalled (since yum
died before the cleanup phase).
Here's the quick-and-dirty command line I used to find the duplicates:
rpm -qa --queryformat='%{NAME}\t%{INSTALLTID}\t%{VERSION}-%{RELEASE}\n' \
| sort \
| awk 'BEGIN{prev=""} {if ($1 == prev) { print prevline; } prev=$1; prevline=$0}' \
| awk '{print $1 "-" $3}' \
| grep -v -e ^kernel -e ^gpg-pubkey-
This will look for any packages that have multiple versions installed, not including anything starting with "kernel" or "gpg-pubkey" (since they normally have more than one version / key installed). For each duplicate package, all but the most recently installed version will be listed.
Running rpm -e --repackage --nodeps
on the package names output by the above took a while, but afterwards, yum update
worked properly again.