“The invoices don’t make sense.
I have no clue what I’m paying for!”
Users struggling to understand what they were paying for cost IBM Cloud $1 million a month in support costs.
Our objective: create a more legible, understandable IBM Cloud bill.
We identified three key drivers for user confusion around their IBM Cloud charges. After releasing the first of four phases to resolve these issues, we saw a reduction in support costs of $250k a month.
How did we uncover those three key drivers?
I structured a discovery research plan which included:
Analyzing billing-related support tickets to identify overarching trends
Interviewing 18 invoice and usage reviewers who had experience with one of the identified trends, 6 per trend. (Sessions were conducted remotely via desktop.)
Cross-disciplinary team members were invited to attend the interviews so they could learn firsthand, leading to faster uptake on resolving the issues at hand.
What did we learn?
Users compare their invoice 1:1 with usage data listed on IBM Cloud’s portal to check for errors and make more informed decisions about optimizing future usage.
Today, they can’t do this because:
[placeholder for support ticket trends analysis visual, here or earlier]
1
Charges listed on your invoice do not match names listed in the usage portal.
“Nowhere on the invoice does the identifier for Storage align to anything. You could be billing me for something that I shouldn't be billed for, but I can't possibly cross-reference this.”- Cloud Operations Coordinator, Enterprise company
2
IBM Cloud billing periods don’t align with our users’ mental model.
IBM Cloud billing periods are matched to a user’s contract date, so a user might have a billing period from October 8th to November 8th.
This is inconsistent with a user’s expectation, which is a standard calendar month: October 1st to October 31st.
To further complicate things, we provided standard month filters in the IBM Cloud portal. So when users matched their charges 1:1 from invoice to usage data in our portal, they had to do this manually.
"With every other cloud vendor I’ve worked with, we normally are billed at the end of the month (or first of the next month), for usage from the previous 30 days."-Chief Innovation Officer, Enterprise company
3
For some services users are billed for usage incurred in a previous month.
IBM invoices for VPC usage 60-90 days in the past. This makes it extremely difficult to plan and manage my cloud expenses in a timely manner. -Chief Innovation Officer, Enterprise company
Evaluating a solution
I presented these findings to our cross-disciplinary team, and they rallied around tackling the issues at hand.
They developed a 4-phase plan to:
Track all usage in one system, which would remove the 60-day charge delays and make all service names consistent (on both the invoice and portal).
Convert all contracts to a standard month (Jan 1-31, etc) so we can charge them on an industry-standard cycle.
Doing this meant taking on 1-2 years worth of work to:
Merge three backend systems
Overhaul our internal process for tracking usage
Change the way we write all new and renewal contracts.
But. Our users can’t wait years.
So, we broke the work into four phases - starting with exploring quick wins which could provide a more streamlined experience by masking the underlying problems.
We started with the assumption that the underlying issues listed above are problematic, but what makes them exceptionally inflammatory is that it can take users ~5-10 hours of support interactions before they are able to pinpoint why they don’t understand what they’re being charged for.
"It took 8 calls with support and one meeting with your team before I understood that these charges were from a few months ago. If I'd known that upfront, I wouldn't have been as shocked by the cost."-DevOps Lead, Enterprise company
We hypothesized that users won’t ask support about their charges if they:
understand that they are being billed for usage prior to that month, without having to leave their invoice.
are able to independently connect services listed on their invoices to those on the portal.
This would not be a replacement for resolving the root cause of these issues, but a first step while we worked towards a more sustainable long-term solution.
The design team came up with an updated invoice concept to iterate on which included things like links to:
How did we iterate?
I structured an evaluative research plan which included:
RITE testing with 5 participants per round, we completed a total of three rounds before launching. 50% had previous experience with one of the issues identified, 50% utilized our competitors and were new to IBM Cloud. (Sessions were conducted remotely via desktop.)
Tasks were based on use cases uncovered in the earlier interviews, to ensure we were testing with real-world problems.
Cross-disciplinary team members were invited to attend the sessions so they could learn firsthand.
What did we learn?
#1 Users needed less detail on their invoice, when it linked directly to the portal.
"If I have all of the metrics available in the portal, I don't need it here too - that just clutters things."-Engineering Lead, Mid-size company
#2 Users expected charges from previous months to be called out separately not only as line items, but also separate categories in the invoice summary section.
"If I have all of the metrics available in the portal, I don't need it here too - that just clutters things."-Engineering Lead, Mid-size company
#2 Users expected charges from previous months to be called out separately not only as line items, but also separate categories in the invoice summary section.
"Linking to details is extremely useful - I wish you'd had this last month when I was making sense of my Kube costs!"-Engineering Lead, Mid-size company