First you need to remove all php5 files :

sudo apt-get remove --purge 'dpkg -l | grep php | grep -w 5.5 | awk '{print $2}' | xargs'

After that try to search for php 5.4. available resources :

apt-cache policy php5

My result :

php5:
Installed: (none)
 Candidate: 5.5.12+dfsg-2+deb.sury.org~saucy+1
 Version table:
 5.5.12+dfsg-2+deb.sury.org~saucy+1 0
 500 http://ppa.launchpad.net/ondrej/php5/ubuntu/ saucy/main amd64 Packages
 5.5.3+dfsg-1ubuntu2.3 0
 500 http://cz.archive.ubuntu.com/ubuntu/ saucy-updates/main amd64 Packages
 500 http://security.ubuntu.com/ubuntu/ saucy-security/main amd64 Packages
 5.5.3+dfsg-1ubuntu2 0
 500 http://cz.archive.ubuntu.com/ubuntu/ saucy/main amd64 Packages

If there is no resources available than we need to add some by following command : 

sudo echo "deb http://php53.dotdeb.org stable all" >> /etc/apt/sources.list

and

apt-get update

If there is a problem with signatures like following :

Reading package lists... Done
W: GPG error: http://php53.dotdeb.org stable Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY xxxxxCODExxxx

we need to sign our Deb repository by commands :

gpg --keyserver keys.gnupg.net --recv-key xxxxxCODExxxx
gpg -a --export xxxxxCODExxxx| sudo apt-key add -
apt-get update

After that execute :

apt-cache policy php5

you should see last result :

5.4.28-1~dotdeb.1 0
500 http://php53.dotdeb.org/ stable/all amd64 Packages
100 /var/lib/dpkg/status

and finally install your version by hitting :

apt-get install php5=5.4.28-1~dotdeb.1

Enjoy 😉