subvolumes are integral to btrfs. You cannot have a layer like luks between them. You can encrypt the whole partition with luks before btrfs or you can encrypt specific directories after btrfs with something like encfs or truecrypt, though doing so loses some of the benefits of btrfs as it can no longer see your individual files.

If you wanted just /home encrypted with luks it would need to be a separate partition (which you could then have btrfs inside with subvolumns on that). Though IMO that gets a bit complicated - I would just opt for encrypting everything (except boot) on the root partition and have one btrfs fs on that partition with as many subvolumes inside that as you like.