It's best not to confuse "Vancouver" with "Vancouver Island" The gardens are in Central Saanich, which is part of Greater Victoria.
Other neat things to see around the south island besides the gardens are Kinsol Trestle (Huge old railway trestle, now used for hikers, cyclists, and horses), Fort Rodd Hill (Historic naval base), Government House (Home of the Lieutenant Governor, or the Queen if she's visiting), The Parliament Buildings, the Royal BC Museum, The Empress Hotel, Beacon Hill Park, loads of Wineries in the Cowichan Valley, the Cowichan River (Floating down it on an inner tube is a popular passtime in the summer, but there are also loads of trails along it too). Lots of hiking trails of varying difficulty, often with some very nice views. The murals in Chemainus and the totem poles (and the world's largest hockey stick) here in Duncan. The Gulf Islands (Small islands between Vancouver Island and the mainland) can also be quite interesting, especially if you like artsy, counter culture, neo-hippy types. The west coast of Vancouver Island, or further north in the central or north island also have interesting stuff. Pacific Rim National Park is great, and Strathcona Provincial Park has alpine meadows that are very nice in the summer. There are also kayak trips and whale watching.