Monthly Archives: March 2015

10.10 – Boot Hangs after Deleting /var/folders Directory

UPDATE 4-13-2015: Apple replied back to my bug report stating it behaves as expected and deleting the /var/folders directory is not supported. So….don’t delete /var/folders.

Previously, in OS 10.9 it has been documented that removing the data in /private/var/folders is a good troubleshooting step for various reasons. There is a good writeup at blog.magnusviri.com describing what /var/folders contains and what issues arise from its bloat. Up thru OS 10.9.5 it hasn’t been an issue deleting contents or the parent directory.

The article above mentions deleting just the contents of /var/folders, which is still good advice. However a haphazard deletion of the /var/folders directory itself will cause issues in 10.10. If you delete the parent folder /var/folders and reboot, the computer will boot to about 50% and hang indefinitely. It will look a lot like the “loginLockout” behavior except the progress bar is at around 50% instead of 33% and the cursor is on screen. Don’t be fooled as I was. This is not loginLockout.

Examining the boot drive while booted from another source showed many errors in system.log when booting about /var/folders/zz not being present. Surprisingly it appears that mkdir is being called but unable to create the directory structure.

.
.
.
Mar 19 09:40:26 macadmins-Mac.local dirhelper[1008]: mkdir(/var/folders/tx): No such file or directory
Mar 19 09:40:26 macadmins-Mac xpcproxy[1007]: libcoreservices: _dirhelper_userdir: 351: stat: /var/folders/tx/56ff45qs2rlc89vggkr8x3yr0000gn/: No such file or directory
Mar 19 09:40:26 macadmins-Mac.local dirhelper[1008]: mkdir(/var/folders/tx): No such file or directory
Mar 19 09:40:26 macadmins-Mac fontworker[1007]: libcoreservices: _dirhelper_userdir: 351: stat: /var/folders/tx/56ff45qs2rlc89vggkr8x3yr0000gn/: No such file or directory
Mar 19 09:40:26 macadmins-Mac fontworker[1007]: libcoreservices: _dirhelper: 454: mkdir: path=/var/folders/tx/56ff45qs2rlc89vggkr8x3yr0000gn/T/ modes[1]=0700: No such file or directory
Mar 19 09:40:26 macadmins-Mac fontworker[1007]: libcoreservices: _dirhelper: 454: mkdir: path=/var/folders/tx/56ff45qs2rlc89vggkr8x3yr0000gn/C/ modes[2]=0700: No such file or directory
Mar 19 09:40:27 macadmins-Mac storeassetd[304]: libcoreservices: _dirhelper: 454: mkdir: path=/var/folders/tx/56ff45qs2rlc89vggkr8x3yr0000gn/T/ modes[1]=0700: No such file or directory
Mar 19 09:40:27 macadmins-Mac fontd[227]: libcoreservices: _dirhelper: 454: mkdir: path=/var/folders/tx/56ff45qs2rlc89vggkr8x3yr0000gn/T/ modes[1]=0700: No such file or directory
Mar 19 09:40:31 macadmins-Mac.local dirhelper[1008]: mkdir(/var/folders/tx): No such file or directory
Mar 19 09:40:31 macadmins-Mac xpcproxy[1011]: libcoreservices: _dirhelper_userdir: 351: stat: /var/folders/tx/56ff45qs2rlc89vggkr8x3yr0000gn/: No such file or directory
Mar 19 09:40:31 macadmins-Mac.local dirhelper[1008]: mkdir(/var/folders/tx): No such file or directory
Mar 19 09:40:31 macadmins-Mac xpcproxy[1012]: libcoreservices: _dirhelper_userdir: 351: stat: /var/folders/tx/56ff45qs2rlc89vggkr8x3yr0000gn/: No such file or directory
Mar 19 09:40:31 macadmins-Mac fontd[227]: libcoreservices: _dirhelper: 454: mkdir: path=/var/folders/tx/56ff45qs2rlc89vggkr8x3yr0000gn/T/ modes[1]=0700: No such file or directory
Mar 19 09:40:31 macadmins-Mac fontd[227]: libcoreservices: _dirhelper: 454: mkdir: path=/var/folders/tx/56ff45qs2rlc89vggkr8x3yr0000gn/T/ modes[1]=0700: No such file or directory
Mar 19 09:40:31 macadmins-Mac loginwindow[64]: libcoreservices: _dirhelper: 454: mkdir: path=/var/folders/tx/56ff45qs2rlc89vggkr8x3yr0000gn/T/ modes[1]=0700: No such file or directory
Mar 19 09:40:31 macadmins-Mac fontd[227]: libcoreservices: _dirhelper: 454: mkdir: path=/var/folders/tx/56ff45qs2rlc89vggkr8x3yr0000gn/T/ modes[1]=0700: No such file or directory
Mar 19 09:40:31 macadmins-Mac.local locationd[241]: Could not write data to disk /var/folders/zz/zyxvpxvq6csfxvn_n00000sm00006d/C/cache.plist
Mar 19 09:40:45 localhost dirhelper[61]: /var/folders/: invalid ownership
.
.
.

To fix the issue I had to maually recreate the directories folders/zz in /var to allow the system to boot once again.

I’ve file a bug report and posted on OpenRadar.

Tagged , ,

Jettisoning the Boot Cache – is it safe?

UPDATE – Apple released OS 10.10.3 on April 8, 2015 that addresses the boot lockout issue described below. Information on the release can be found at Apple’s support site.

You may have heard of this little problem with Yosemite some are giving the moniker “Login Lockout”. The problem, as described at AFP548.com, is that the computer will boot about 1/3 of the way and halt. It happens randomly, but more often after a hard reboot.

A community fix has been gaining traction at JAMFNation that suggests editing /etc/rc.server to run /usr/sbin/BootCacheControl jettison that seems to work. But is it safe to jettison the boot cache every boot?

In short, probably.

My friend @mikeymikey on Twitter spelunked into the issue at depths I’m not comfortable with and discovered, among other things, that the BootCache.kext has code that calls for a jettison on multiple occasions. The code is open sourced and available to browse at Apple’s open source site. Searching for “jettison” in the code will reveal some nice comments about different scenarios that would invalidate the cache. You can also see the command’s semi-hidden documentation by running /usr/sbin/BootCacheControl without an option. It will display the following, jettison being one option.


Usage: /usr/sbin/BootCacheControl [-vvv] [-f ] start
           Start recording history and play back .
       /usr/sbin/BootCacheControl [-vvv] [-f ] stop
           Stop recording history and write the playlist to .
       /usr/sbin/BootCacheControl mount
           Notify the boot cache of a new mount.
       /usr/sbin/BootCacheControl [-vvv] jettison
           Jettison the cache.
       /usr/sbin/BootCacheControl statistics
           Print statistics for the currently-active cache.
       /usr/sbin/BootCacheControl tag
           Insert the end-prefetch tag.
       /usr/sbin/BootCacheControl [-vvv] [-t batchnum] -f  merge  [...]
           Merge ... into .
           Playlist files after the first will be offset  batches, if provided
       /usr/sbin/BootCacheControl [-c] -f  print
           Print the contents of .
       /usr/sbin/BootCacheControl -f  unprint
           Read a playlist from standard input and write to .
       /usr/sbin/BootCacheControl -f  generate []
           Generate a playlist from standard input data for  and write to .
       /usr/sbin/BootCacheControl -f  truncate 
           Truncate  to  entries.
       /usr/sbin/BootCacheControl -f  generalize
           Modify  to apply to any root volume.

The result of running this on every boot just means that the computer may boot a smidge slower as the cache is not there to reference.

There are rumors of a fix coming from Apple but it’s unclear if that will be included in 10.10.3 or not. Adding the jettison to rc.server appears to be a valid workaround until the official fix is out. This temporary workaround is easily revertible by just removing the rc.server file and rebooting.

Tagged , ,

Custom location detection using CIDR range

Ask the right questions and you’ll get helpful answers.  Ask some code if an IP exists within the CIDR range and the brief answer can be powerful.

Services like munki, reposado, or other internal offering in your company can be replicated out to other locations but if the client settings never change they’ll continue to use the original settings even if the computer is in a different office across the country. That can be expensive in time and bandwidth — if there is inter-office connectivity at all. Every computer on the network gets an IP, and every IP is in a CIDR range. Why not use that information to help.

You can determine where a computer is based on your documented CIDR ranges and checking if the current IP is in one of those ranges. Systematically processing that information can help with the task of re-configuring based on locale. A simple boolean answer can be parsed to mean anything for your environment. It’s what you do with that answer that makes the difference.

First, you need to know what CIDR ranges are active in your company’s network.  Whoever setup the network should be able to provide that information.  Be sure to double check the translation of CIDR to IP ranges. There is a site that can translate CIDR to the IP range. I had a few instances of CIDRs provided were overlapping and caused unexpected results when certain IPs were used.

Next, figure out what information you want returned.  For my implementation I went the simple route and have the script return a single value that signifies which company office the IP is in.  You can have it return as much as you want and make it as complex as you want.

The following perl code returns true if a supplied IP exists in a supplied CIDR range.

#!/usr/bin/perl
use Net::CIDR::Lite;
use Getopt::Std;

#"i"P address and "c"IDR notated range
getopt ('ic');

my $cidr = Net::CIDR::Lite->new;
$cidr->add($opt_c);
print "true\n" if $cidr->find($opt_i);

The following script iterates thru the known company CIDR ranges with the current computer’s IP address and sends those two pieces of information to the perl script until a match is found. When the IP exists inside a site’s CIDR range it will return the value of the site.

#!/bin/bash
# To use this in other scripts reference it by
#   /Library/Management/Scripts/cidr/cidr-check.sh [ -i ip_address ]
#
#   If no IP is provided by the -i argument the current IP on the computer the script is being run on will be used.
# 
#   Site            Return Value
#
#   Des Moines      DSM    
#   New York        NY
#   Chicago         CHI
###############################################################

#Locations of helper scripts
scriptLocation=/Library/Management/Scripts/cidr/
cidrCheckScript=check_IP_and_CIDR.pl

activeiface=`route get google.com | awk '/interface/ {print $2}'`

#Get the current IP address
#Read in the arguments
usage="Usage: cidr-check.sh [-h] [-i ip_address]"
while getopts i:h flag
do case "$flag" in
	i ) ip="$OPTARG";;
	h ) echo $usage
		exit 1;;
    * ) echo $usage
		exit 1;;
esac
done

#Shift out the switches and arguments, leaving the actual
#variables we're operating against
shift $(($OPTIND - 1))

if [ -z $ip ]
then
    ip=`/sbin/ifconfig $activeiface | /usr/bin/awk '/inet / {print $2}'`
fi

#Flag to mark whether the IP has been found
ipFound=n

###############################################################
#           Lists of CIDR range names                         #
###############################################################
dsmCidrRangeNames=(currentCIDR_DSM1 currentCIDR_DSM2)
dsmCidrRangeNames_Site="DSM"
chiCidrRangeNames=(currentCIDR_CHI)
chiCidrRangeNames_Site="CHI"
nycCidrRangeNames=(currentCIDR_NYC)
nycCidrRangeNames_Site="NY"

##Eastern Time CIDR ranges
    #New York
        currentCIDR_NYC=1.2.0.0/18
##Central Time CIDR ranges
    #Chicago
        currentCIDR_CHI=1.3.224.0/20
    #Des Moines
        currentCIDR_DSM1=1.4.0.0/16
        currentCIDR_DSM2=1.5.0.0/16
###############################################################

if [ "$ipFound" = "n" ]
then
	for cidrRange in ${dsmCidrRangeNames[@]}
	do
		if [ "`${scriptLocation}${cidrCheckScript} -i "$ip" -c ${!cidrRange}`" = "true" ]
		then
			returnValue="${dsmCidrRangeNames_Site}"

			#Mark that we fnd the IP address
			ipFound=y

			#Break out of the for loop
			break
		fi
	done
fi

#CHI
if [ "$ipFound" = "n" ]
then
	for cidrRange in ${chiCidrRangeNames[@]}
	do
		if [ "`${scriptLocation}${cidrCheckScript} -i "$ip" -c ${!cidrRange}`" = "true" ]
		then
			returnValue="${chiCidrRangeNames_Site}"

			#Mark that we fnd the IP address
			ipFound=y

			#Break out of the for loop
			break
		fi
	done
fi

#NYC
if [ "$ipFound" = "n" ]
then
	for cidrRange in ${nycCidrRangeNames[@]}
	do
		if [ "`${scriptLocation}${cidrCheckScript} -i "$ip" -c ${!cidrRange}`" = "true" ]
		then
			returnValue="${nycCidrRangeNames_Site}"

			#Mark that we fnd the IP address
			ipFound=y

			#Break out of the for loop
			break
		fi
	done
fi

if [ "$ipFound" = "n" ]
then
	#We didn't find the IP in a CIDR range
	/bin/echo "IP: $ip not within the specified CIDR ranges." 2>&1
	exit -1
fi

/bin/echo "${returnValue}"

exit 0

With that information I now know what location that computer has an IP from. Those two pieces of code can be re-used as a function in any number of ways. One way we use it is to configure munki to point at the local repo for pkgs, icons, and client_resources during its preflight execution on every run. That way regardless of what office the computer is in it won’t cross the WAN to pull a pkg.

#!/bin/bash
# Script to determine local site and change munki urls to local resources
# Uses 'cidr-check.sh' to determine the site based on IP

#Static variables
munkiConfig=/Library/Preferences/ManagedInstalls
cidrCheckScript=/Library/Management/Scripts/cidr/cidr-check.sh
centralMunkiServer=munki.YOUR.DOMAIN.COM
#################

#Check to see if we're running as root
if [ `id -u` != 0 ]; then
	echo $0: "You must have root privileges to run this command!  Exiting."
	exit 1
fi

#Set the variable "localSite" to the returned value from 'cidr-check.sh'
localSite=`"${cidrCheckScript}"`
echo Local site determined to be "${localSite}"

#Determining the local PackageURL to use
case $localSite in

	DSM )	
			localServer=('munki-dsm.YOUR.DOMAIN.COM')
			;;

	NY )	
			localServer=('munki-nyc.YOUR.DOMAIN.COM')
			;;

	CHI )	
			localServer=('munki-chi.YOUR.DOMAIN.COM')
			;;
	#All other locations should default to the central server
	* )		
			localServer=('munki-dsm.YOUR.DOMAIN.COM')
			;;
esac

echo Local server determined to be "${localServer}"

#If we can't talk to the local munki server, use the central server
if ! ping -o -c 5 "${localServer}" > /dev/null 2>&1
then
	localServer="${centralMunkiServer}"
fi

#Write the local munki URLs
defaults write "${munkiConfig}" PackageURL http://"${localServer}"/repo/pkgs
defaults write "${munkiConfig}" IconURL http://"${localServer}"/repo/icons
defaults write "${munkiConfig}" ClientResourceURL http://"${localServer}"/repo/client_resources

#Write the central munki URL for everything else
defaults write "${munkiConfig}" SoftwareRepoURL http://"${centralMunkiServer}"/repo

#echo Converting ManagedInstalls.plist to XML, the original format of the file
plutil -convert xml1 ${munkiConfig}.plist

I’m sure there are multiple ways to handle this type of problem. I’d love to hear how you handle the complexity of roaming computers and keeping configurations local.

Tagged , ,