billtest1 WordPress workflow

This is all about improving the way we develop WordPress sites. And No More Cowboy Coding gives a good intro to the basic concepts and why they make sense. (But don’t use this as a step-by-step guide, because in our case we’re using different elements to put it all together.)

The local set-up on your Mac

First up, we need a way of testing a site on a local server – but ideally one that matches the live server. This is where Virtual Machines come in.

This article describes the key bits of the local server set-up. At the heart of it is VVV (Varying Vagrant Vagrants), which sets up a virtual machine on a local computer and provisions it with everything needed to act as a WordPress server.

Everything – from the operating system up!

Half this stuff we’ll probably never use – but it’s reassuring to know it’s there, and great to be able to fine tune the server without fear of breaking anything that matters.

Virtual machines are completely isolated from the host machine they run on – apart from a few shared folders that are mirrored on both machines and whose contents are kept in synch. This allows us to use Dreamweaver as usual, but without having to “upload” anything from the host Mac to the Vagrant Box (as this particular VM is sometimes called).

Getting started

Go to The First Vagrant Up at VVV and work through the list.
Install this dashboard.
Install the VVV Site Wizard: Variable VVV.

Dealing with the Uploads folder

Thanks to this article, we don’t have to copy all the media library images. Instead, if something’s missing, we can just load it from the live site. Simples.

In vagrant-local/config/nginx-config/sites there should be a .conf file for each site. Towards the end, these files should contain something like…

server {

    # ...various bits and pieces...
    
    # Dealing with missing images
    # Directives to send expires headers and turn off 404 error logging.
    location ~* \.(png|jpe?g|gif|ico)$ {
        expires 24h;
        log_not_found off;
        try_files $uri $uri/ @production;
    }

    location @production {
        resolver 8.8.8.8;
        proxy_pass http://www.leedstidal.org/$uri;
    }

}

NB: Watch out for sub-folders in path! Use Firebug to examine an uploads image on the live site.

Tracking changes across different servers

This is the other vital element – version control – keeping track of all the changes via a Git repository. This article describes fairly closely the system that I’m experimenting with. And there’s some excellent stuff in the comments, too: I found this one really helpful in getting my head round where the definitive versions of files and database should live.

Command line processes

To create a new database. open database/init-custom.sql and add something like:

CREATE DATABASE IF NOT EXISTS `dev_s3word`; GRANT ALL PRIVILEGES ON `dev_s3word`.* TO 'dev_s3word'@'localhost' IDENTIFIED BY 'dev_s3word';

To import sql into this blank database, use Sequel Pro File > Import.

Cloning a repository
Connecting to MySQL in the Vagrant box

Set up SSH keys for Bitbucket

Vital links

Setting up a WordPress VVV Vagrant Workflow
This article describes fairly closely the set-up that I’m using for the local server on my Mac.

Git and WordPress workflow for Local Development to Staging Webserver
This article describes a system where pushing directly to the repo on the remote server transfers all the files to a separate working directory.

http://gitstart.tinkerbill.co.uk/
http://ryanstutorials.net/linuxtutorial/
https://github.com/Varying-Vagrant-Vagrants/VVV
https://github.com/TinkerBill/vagrant-2
http://readwrite.com/2013/09/30/understanding-github-a-journey-for-beginners-part-1
https://support.krystal.co.uk/entries/23161956-How-to-use-public-private-keys-for-SSH-and-SFTP-Mac-OS-X
https://www.digitalocean.com

SitePoint book: Jump Start PHP Environment

SitePoint book: Jump Start Git

Cloning to production server

Seems to work best with https:// rather than ssh://.

jesse-lewiss-iMac-2:~ bill$ hera
tinkerbi@hera [~]# cd ~/public_html/tidal
tinkerbi@hera [~/public_html/tidal]# git clone -b master --single-branch https://TinkerBill@bitbucket.org/TinkerBill/tidal.git tidal_git_live
Cloning into 'tidal_git_live'...
Password for 'https://TinkerBill@bitbucket.org': 
remote: Counting objects: 1298, done.
remote: Compressing objects: 100% (1208/1208), done.
remote: Total 1298 (delta 68), reused 1298 (delta 68)
Receiving objects: 100% (1298/1298), 5.74 MiB | 1.54 MiB/s, done.
Resolving deltas: 100% (68/68), done.
Checking out files: 100% (1194/1194), done.

How to move/ separate the .git folder out of your working tree

A neat thing about git init is that you can run it with the --separate-git-dir switch on an existing working tree whose repository is within it, and it will move the repository out to wherever you specify and do the linking.

tinkerbi@hera [~/public_html/tidal]# cd tidal_git_live
tinkerbi@hera [~/public_html/tidal/tidal_git_live]# git init --separate-git-dir ~/private/tidal/tidal.git
Reinitialized existing Git repository in /home/tinkerbi/private/tidal/tidal.git/

Enabling SSH access to Krystal

From Krystal FAQ: Do you allow SSH Shell access?:

Click ‘Web hosting’.
Click ‘Manage’ next to the cPanel plan you wish to enable SSH for.
On the right hand side click the ‘SSH’ option from the ‘Management’ menu.
Read the advice and click the green ‘Got it, enable SSH for (cPanel user)’ button.
Please wait whilst this is enabled. Once done a green bar saying ‘SSH is currently enabled for your account’ will appear.
Your log in details will now show.
Please remember to disable SSH from here also when it is no longer needed.

Setting up a public key on a Krystal server

From How to use public/private keys for SSH and SFTP – Mac OS X.

In Terminal, enter key, an alias which copies the tinkerbill public key to the clipboard.
(Assumes that (a) you’ve set up this alias and (b) you’ve got an appropriately named public key.)
Log into your cPanel account, and click on the SSH/Shell Access icon in the Security group of icons. Then click the Manage SSH keys button.
Paste your public key into the Paste the Public Key in this box: text box. Give your public key a name (tinkerbill), and click the Import button. If you don’t give your key a name, it will default to being called id_dsa.
Back on the Manage SSH Keys cPanel screen, click Manage Authorization for your key.
Then, on the Manage Authorization screen just click the Authorize button and you should receive a confirmation that the key has been authorised.

Making a git repository out of the existing live site

From A web-focused Git workflow

jesse-lewiss-iMac-2:~ bill$ ss
Which site?
l4c
leedsfor@poseidon [~]# cd public_html/live

leedsfor@poseidon [~/public_html/live]# git init
Initialized empty Git repository in /home/leedsfor/public_html/live/.git/

leedsfor@poseidon [~/public_html/live]# git add .

leedsfor@poseidon [~/public_html/live]# git config --global user.email "bill.phelps@ntlworld.com"
leedsfor@poseidon [~/public_html/live]# git config --global user.name "Bill Phelps"

leedsfor@poseidon [~/public_html/live]# git commit -m"initial import of pre-existing web files"
[master (root-commit) ce82167] initial import of pre-existing web files
 1168 files changed, 205380 insertions(+)
..... lots of files ......

leedsfor@poseidon [~/public_html/live]# git init --separate-git-dir ~/private/live.git
Reinitialized existing Git repository in /home/leedsfor/private/live.git/

jesse-lewiss-iMac-2:~ bill$ ss
Which site?
l4c
leedsfor@poseidon [~]# cd private/live.git
leedsfor@poseidon [~/private/live.git]# git remote add origin git@bitbucket.org:TinkerBill/l4c.git
leedsfor@poseidon [~/private/live.git]# git push -u origin --all
The authenticity of host 'bitbucket.org (104.192.143.2)' can't be established.
RSA key fingerprint is 97:8c:1b:f2:6f:14:6b:5c:3b:ec:aa:46:46:74:7c:40.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'bitbucket.org,104.192.143.2' (RSA) to the list of known hosts.
Counting objects: 1469, done.
Delta compression using up to 32 threads.
Compressing objects: 100% (1368/1368), done.
Writing objects: 100% (1469/1469), 6.33 MiB | 2.68 MiB/s, done.
Total 1469 (delta 157), reused 0 (delta 0)
To git@bitbucket.org:TinkerBill/l4c.git
 * [new branch]      master -> master
Branch master set up to track remote branch master from origin.

leedsfor@poseidon [~/private/live.git]# git push -u origin --tags # pushes up any tags
Everything up-to-date


jesse-lewiss-iMac-2:vagrant-2 bill$ vv create
New VVV Site Setup 
 Name of new site directory: l4c
 Domain to use (leave blank for l4c.dev): local.l4c
 WordPress version to install (leave blank for latest version or trunk for trunk/nightly version): 
 Install as multisite? (y/N): N
 Git repo to clone as wp-content (leave blank to skip): 
 Local SQL file to import for database (leave blank to skip): 
 Remove default themes and plugins? (y/N): N
 Add sample content to site (y/N): N
 Enable WP_DEBUG and WP_DEBUG_LOG (y/N): y

About to perform the following: 
* Halt Vagrant (if running)
* Create directory l4c in /Users/bill/VirtualBox VMs/vagrant-2/www
* Create files vvv-init.sh, wp-cli.yml, and vvv-hosts in directory l4c
* Create file l4c.conf in /Users/bill/VirtualBox VMs/vagrant-2/config/nginx-config/sites
* Run `vagrant up --provision` to initialize site

Provisioning Vagrant will do the following: 
* Create database l4c
* Install WordPress (release version) in the htdocs directory
* Make the site visible at local.l4c

 
 Continue (y/n)? : y
[Success] New VVV setup starting for site 'l4c' 

... lot of stuff ...

[Success] New VVV Site Setup: Done! 
Directory: /Users/bill/VirtualBox VMs/vagrant-2/www/l4c
URL:       local.l4c
Username:  admin
Password:  password

[Process completed]


... then do a lot of manual tweaking on files and ...


jesse-lewiss-iMac-2:~ bill$ cdv2
jesse-lewiss-iMac-2:vagrant-2 bill$ vagrant provision

... then use SourceTree to clone into an empty l4c_htdocs folder ...

Files that need tweaking

database/init-custom.sql
www/sitename/vvv-hosts // local.sitename
www/sitename/vvv-init.sh // see above
www/sitename/wp-cli.yml // path: sitename_htdocs
private/sitename/wp-local-settings.php
config/nginx-config/sites/sitename.conf

The Gitflow Workflow

This Atlassian tutorial explains the thinking behind the workflow that Dave’s set up for Carplus. It makes a lot of sense for bigger sites where a number of people are collaborating. And probably for any site where you want to be able to show the client changes online without disturbing the live site.

Troubleshooting

Locked files

Git relies on having files unlocked (ie: not checked in), otherwise many operations fail, with errors like

error: unable to unlink old 'wp-content/themes/carplus_2015/bbpress/user-subscriptions.php' (Operation not permitted)

In DreamWeaver local view, ctrl-clicking a folder allows you to Unlock all the files within it.

Restarting a Vagrant box

The Variable VVV site creation system will reload WordPress if it can’t recognise the file structure. As it stands, changing to the standard Wingfinger cms folder structure triggers a reload the next time the VM is rebuilt. The clue is that suddenly you have a lot of uncommitted changes in your working tree!

Temporary solution is simply to delete them all. Bit of a pain… Working on it…

Whenever vagrant does a rebuild or provision, it follows the script at vagrant-local/provision/provision.sh which looks for and runs every script it can find called vvv-init.sh – the extension is required. These are initially created by the Variable VV script at vagrant-local/www/default/vv/vv – you can find the relevant lines by searching for Installing WordPress.

Until we rewrite that code, we’ll need to edit each vvv-init.sh file to look something like this:

if [ ! -d "l4c_htdocs/wp-admin" ] && [ ! -d "l4c_htdocs/cms/wp-admin" ]; then
	echo 'Installing WordPress (release version) in l4c/l4c_htdocs...'
	if [ ! -d "./l4c_htdocs" ]; then
		mkdir ./l4c_htdocs
	fi
	cd ./l4c_htdocs
	wp core download --locale=en_US --allow-root 
	wp core config --dbname="l4c" --dbuser=wp --dbpass=wp --dbhost="localhost" --dbprefix=wp_ --locale=en_US --allow-root --extra-php <<PHP
define('WP_DEBUG', true);
define('WP_DEBUG_DISPLAY', false);
define('WP_DEBUG_LOG', true);
define('SCRIPT_DEBUG', true);
define('JETPACK_DEV_DEBUG', true);
PHP
	wp core install --url=local.l4c --title="l4c" --admin_user=admin --admin_password=password --admin_email=admin@localhost.dev --allow-root
	
	
  
  
	
	
	cd -
else
	echo 'Found wp-admin - no WordPress install required'

fi

Google breaks the Interweb!

Chrome & Firefox now force .DEV domains to HTTPS.

So any .dev web domain (and several others too) result in impassable security errors – even on local VMs. Grrr.

In particular, this meant no access to the VVV Dashboard at http://vvv.dev. Resolved this by replacing vvv.dev with vvv.loc in the following files:

/Users/bill/VirtualBox VMs/vagrant-2/www/vvv-hosts
/Users/bill/VirtualBox VMs/vagrant-2/config/nginx-config/sites/default.conf
/Users/bill/VirtualBox VMs/vagrant-2/config/wp-cli/local.php
/Users/bill/VirtualBox VMs/vagrant-2/www/default/dashboard/functions.php
/Users/bill/VirtualBox VMs/vagrant-2/www/default/dashboard/partials/navbar.php

Major upheavals Sept 2018

See Updating VVV etc – which has now been copied across to the remainder of this page…

Background Sept 2018

First started playing with GIT etc around Dec 2015.
Original: Vagrant 1.7.4, Virtualbox 5.0.10, VVV 1.2.0
Upgrading to: Vagrant 2.1.5, Virtualbox 5.2.18, VVV 2.2.1
——

Feb 2019

Upgrading to: Vagrant 2.2.3, Virtualbox 6.0.4, VVV 2.5.1
——

Links

Main docs are at https://varyingvagrantvagrants.org/
http://lance.bio/2017/11/16/using-custom-vagrant-sites-wordpress-development/
https://tomjn.com/2017/01/16/whats-new-vvv-2-0/

Started with https://varyingvagrantvagrants.org/docs/en-US/installation/keeping-up-to-date/ but hit errors with hostsupdater.

jesse-lewiss-iMac-2:~ bill$ cdv2
jesse-lewiss-iMac-2:vagrant-2 bill$ vagrant plugin uninstall vagrant-triggers
WARNING: Vagrant has detected the `vagrant-triggers` plugin. This plugin conflicts
with the internal triggers implementation. Please uninstall the `vagrant-triggers`
plugin and run the command again if you wish to use the core trigger feature. To
uninstall the plugin, run the command shown below:

  vagrant plugin uninstall vagrant-triggers

Note that the community plugin `vagrant-triggers` and the core trigger feature
in Vagrant do not have compatible syntax.

To disable this warning, set the environment variable `VAGRANT_USE_VAGRANT_TRIGGERS`.
Uninstalling the 'vagrant-triggers' plugin...
Bundler, the underlying system Vagrant uses to install plugins,
reported an error. The error is shown below. These errors are usually
caused by misconfigured plugin installations or transient network
issues. The error from Bundler is:

Unable to resolve dependency: user requested 'vagrant-hostsupdater (> 0)'

[Process completed]

jesse-lewiss-iMac-2:/ bill$ cdv2
jesse-lewiss-iMac-2:vagrant-2 bill$ vagrant -v
Vagrant 2.1.5
jesse-lewiss-iMac-2:vagrant-2 bill$ vagrant box update
==> default: Box 'ubuntu/trusty64' not installed, can't check for updates.

Tried this: https://github.com/Varying-Vagrant-Vagrants/VVV/issues/1258#issuecomment-386526612 and this: https://github.com/cogitatio/vagrant-hostsupdater/issues/164#issuecomment-410534909

Backed up vagrant-2 folder to Bill’s work.

1. vagrant destroy

2. download Vagrant 2.1.5

3. run uninstaller

4. remove ~/.vagrant.d (from user: couldn’t find in root)
cd ~ rm -rf .vagrant.d

5. download VirtualBox 5.2.18

6. run uninstaller

7. install Vagrant

8. install VirtualBox

9. Then use git to upgrade VVV to 2.2.1 – see https://varyingvagrantvagrants.org/docs/en-US/installation/convert-to-git/
NB: original VVV doesn’t have either vvv-config.yml or vvv-custom.yml.

10. Then from https://varyingvagrantvagrants.org/docs/en-US/troubleshooting/:

# Turn the machine on (so destroy can run its cleanup)
vagrant up
# Destroy the machine
vagrant destroy
# Make sure we use the latest version of the base box
vagrant box update
# Make sure the recommended vagrant plugins are installed
vagrant plugin install vagrant-hostsupdater
vagrant plugin uninstall vagrant-triggers
# And that they're all up to date
vagrant plugin update
# Start VVV and create the VM from scratch
vagrant up --provision

Feb 2019

Trouble. Restarted Mac then vagrant up didn’t work. Reloaded VirtualBox & Vagrant. Eventually got it running – but then mcp.loc wouldn’t connect to its wordpress database.

Problem seemed to be password. Using root root allowed access to phpMyAdmin (and Sequel Pro tunnel), but the specific password (in wp-local-settings.php, vvv-init.sh and init-custom.sql) for tinkerbi_metausr wouldn’t allow access to database tinkerbi-metacp.

Solution was…

https://help.ubuntu.com/community/MysqlPasswordReset
http://www.grahamwideman.com/gw/tech/mysql/perms/index.htm

jesse-lewiss-iMac-2:~ bill$ cdv2
jesse-lewiss-iMac-2:vagrant-2 bill$ vagrant ssh
__ __ __ __ 
\ V\ V\ V / Varying Vagrant Vagrants
 \_/\_/\_/  Have a nice day 
 

vagrant@vvv:~$ mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 212
Server version: 10.3.12-MariaDB-1:10.3.12+maria~trusty mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> \h

General information about MariaDB can be found at
MariaDB Foundation


List of all MySQL commands:
Note that all text commands must be first on line and end with ';'
?         (\?) Synonym for `help'.
clear     (\c) Clear the current input statement.
connect   (\r) Reconnect to the server. Optional arguments are db and host.
delimiter (\d) Set statement delimiter.
edit      (\e) Edit command with $EDITOR.
ego       (\G) Send command to mysql server, display result vertically.
exit      (\q) Exit mysql. Same as quit.
go        (\g) Send command to mysql server.
help      (\h) Display this help.
nopager   (\n) Disable pager, print to stdout.
notee     (\t) Don't write into outfile.
pager     (\P) Set PAGER [to_pager]. Print the query results via PAGER.
print     (\p) Print current command.
prompt    (\R) Change your mysql prompt.
quit      (\q) Quit mysql.
rehash    (\#) Rebuild completion hash.
source    (\.) Execute an SQL script file. Takes a file name as an argument.
status    (\s) Get status information from the server.
system    (\!) Execute a system shell command.
tee       (\T) Set outfile [to_outfile]. Append everything into given outfile.
use       (\u) Use another database. Takes database name as argument.
charset   (\C) Switch to another charset. Might be needed for processing binlog with multi-byte charsets.
warnings  (\W) Show warnings after every statement.
nowarning (\w) Don't show warnings after every statement.

For server side help, type 'help contents'

MariaDB [(none)]> use tinkerbi_metacp
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [tinkerbi_metacp]> show databases;
+----------------------+
| Database             |
+----------------------+
| billtest1            |
| chapelto_wp15        |
| information_schema   |
| mysql                |
| performance_schema   |
| tinkerbi_metacp      |
| wordpress_unit_tests |
| wordpressdefault     |
+----------------------+
8 rows in set (0.001 sec)

MariaDB [tinkerbi_metacp]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.002 sec)

MariaDB [tinkerbi_metacp]> SET PASSWORD FOR tinkerbi_metausr@'localhost' = PASSWORD('thepassword');
Query OK, 0 rows affected (0.001 sec)

MariaDB [tinkerbi_metacp]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.001 sec)

MariaDB [tinkerbi_metacp]> \q
Bye
vagrant@vvv:~$

And then it worked!

Problem not helped by failure of Sequel Pro to allow changes to Users tab. See https://github.com/sequelpro/sequelpro/issues/3167

May 2020

As above, plus…

jesse-lewiss-iMac-3:~ bill$ cdv2
jesse-lewiss-iMac-3:vagrant-2 bill$ vagrant ssh -c "db_backup"
Running Custom Vagrant file with additional vagrant configs at /Users/bill/VirtualBox VMs/vagrant-2/Customfile

Finished running Custom Vagrant file with additional vagrant configs, resuming normal vagrantfile execution

Performing Database Backups
Backing up Database chaco...
Database chaco backed up...
Backing up Database chapelto_wp15...
Database chapelto_wp15 backed up...
Backing up Database tinkerbi_metacp...
Database tinkerbi_metacp backed up...
Backing up Database wordpress-default...
Database wordpress-default backed up...
Backing up Database wordpressdefault...
Database wordpressdefault backed up...
Backing up Database wordpressdevelop...
Database wordpressdevelop backed up...
Connection to 127.0.0.1 closed.
jesse-lewiss-iMac-3:vagrant-2 bill$

WordPress workflow

The local set-up on your Mac

Getting started

Dealing with the Uploads folder

Tracking changes across different servers

Command line processes

Vital links

Cloning to production server

Enabling SSH access to Krystal

Setting up a public key on a Krystal server

Making a git repository out of the existing live site

Files that need tweaking

The Gitflow Workflow

Troubleshooting

Locked files

Restarting a Vagrant box

Google breaks the Interweb!

Major upheavals Sept 2018

Background Sept 2018

Feb 2019

Links

Feb 2019

May 2020

Pages

Archives

Categories

Meta