Denormalization is normal with the Firebase Database - The Firebase Database For SQL Developers #6

By: Firebase

375   10   19847

Uploaded on 12/22/2016

Check out our old but still good blog post: https://goo.gl/DoU2t4

Welcome to the sixth video in the Firebase Database for SQL Developers series!

Coming from a SQL background you likely know plenty about normalization. But come prepared to learn all about denormalization! Denormalization is the process of duplicating data in order to reduce or simplify querying. While it may be strange coming from a SQL background, it's a common practice in NoSQL databases that will make reads more performant and your life much easier.

Watch more videos from this series: https://goo.gl/ZDcO0a

Subscribe to the Firebase Channel: https://goo.gl/9giPHG

Comments (17):

By anonymous    2017-09-20

I had the exact similar problem retrieving users by IDs. I found that there was no way of making the IN query in Firebase.

David suggests in his video either denormalizing the database or making a query for every id.

I did not like denormalizing so I queried every user. If you're in to RxJava, here's what I did

@NonNull
private Observable<List<User>> userIdsToUsers(@NonNull final List<String> uids) {
    if (uids.isEmpty()) {
        return Observable.just(new ArrayList<>());
    }

    return Observable.combineLatest(userIdsToUserObservables(uids),
            new ListCastFunc<>(User.class));
}

@NonNull
private List<Observable<User>> userIdsToUserObservables(@NonNull final List<String> uids) {
    final List<Observable<User>> result = new ArrayList<>();
    for (final String uid : uids) {
        result.add(userIdToUserObservable(uid));
    }
    return result;
}

@NonNull
private Observable<User> userIdToUserObservable(@NonNull final String uid) {
    return RxFirebaseDatabase.observeValueEvent(getUsersReference().orderByKey().equalTo(uid))
            .flatMap(this::dataSnapshotToUserObservable);
}


@NonNull
private Observable<User> dataSnapshotToUserObservable(@NonNull final DataSnapshot ds) {
    return Observable.fromCallable(() -> dataSnapshotToUser(ds));
}

@Nullable
private static User dataSnapshotToUser(@NonNull DataSnapshot ds) {
    if (!ds.exists()) {
        return null;
    }

    return parsedObjHere;
}

private static final class ListCastFunc<T> implements FuncN<List<T>> {

    @NonNull
    private final Class<T> targetClass;

    ListCastFunc(@NonNull final Class<T> targetClass) {
        this.targetClass = targetClass;
    }

    @Override
    public List<T> call(final Object... args) {
        if (args == null || args.length == 0) {
            return new ArrayList<>();
        }
        final List<T> results = new ArrayList<>(args.length);
        for (final Object arg : args) {
            if (arg != null && targetClass.isAssignableFrom(arg.getClass())) {
                //noinspection unchecked
                results.add((T) arg);
            }
        }
        return results;
    }
}

Original Thread

By anonymous    2017-09-20

There is no way to ignore child nodes when using onWrite(). Use denormalization to trigger the function on just the data you require. Check out these resources to find out more:

YouTube - Denormalization is Normal with the Firebase Database

Firebase Blog Post - Denormalizing your Data is Normal

Original Thread

By anonymous    2017-09-20

The fact that mongoDB is not relational have led some people to consider it useless. I think that you should know what you are doing before designing a DB. If you choose to use noSQL DB such as MongoDB, you better implement a schema. This will make your collections - more or less - resemble tables in SQL databases. Also, avoid denormalization (embedding), unless necessary for efficiency reasons.

If you want to design your own noSQL database, I suggest to have a look on Firebase documentation. If you understand how they organize the data for their service, you can easily design a similar pattern for yours.

As others pointed out, you will have to do the joins client-side, except with Meteor (a Javascript framework), you can do your joins server-side with this package (I don't know of other framework which enables you to do so). However, I suggest you read this article before deciding to go with this choice.

Edit 28.04.17: Recently Firebase published this excellent series on designing noSql Databases. They also highlighted in one of the episodes the reasons to avoid joins and how to get around such scenarios by denormalizing your database.

Original Thread

By anonymous    2017-09-23

This is the best option for solving your problem. Unfortunately, with your actual database structure you cannot achieve this. Pelase note, that [Denormalization is normal with the Firebase Database](https://www.youtube.com/watch?v=vKqXSZLLnHA). See this video and you'll understand better. As a conclusion, duplicating data is for simplify and reduce query and this what you need. ;)

Original Thread

By anonymous    2017-10-01

Then create another mode in which you can add only the desired data and then attach a listener only at that locatin. This is called denormalization. To a better understanding, please see [Denormalization is normal with the Firebase Database](https://www.youtube.com/watch?v=vKqXSZLLnHA)

Original Thread

By anonymous    2017-10-08

There is no problem within Firebase to have millions of user nodes. It is true that in Firebase in not a good practice to download the entire user node because it would be a waste of bandwith and performance but there is a method with which you can solve this problem and is called `denormalization`. For this i recomand you see the following tutorial for a better understanding: [Denormalization is normal with the Firebase Database](https://www.youtube.com/watch?v=vKqXSZLLnHA). So you don't need to be worried about this.

Original Thread

By anonymous    2017-11-27

There are good answers here but because you have asked me, i'll try to give you some more details. The best practice for Implementing In-app Billing is the official documentation. Please se below my my answer for some of your questions.

As many other users have advised you, so do I, use Firebase as your database for your app. It's a NoSQL database and it's very easy to use.

How do you store the product?

Please see below a database structure that i recomand you to use for your app.

Firebase-root
    |
    --- users
    |     |
    |     --- uid1
    |          |
    |          --- //user details (name, age, address, email and so on)
    |          |
    |          --- products
    |                |
    |                --- productId1 : true
    |                |
    |                --- productId2 : true
    |
    --- products
    |     |
    |     --- productId1
    |     |     |
    |     |     --- productName: "Apples"
    |     |     |
    |     |     --- price: 11
    |           |
    |           |
    |           --- users
    |                |
    |                --- uid1: true
    |                |
    |                --- uid2: true
    |
    --- purchasedProducts
         |      |
         |      --- uid1
         |           |
         |           --- productId1: true
         |           |
         |           --- productId2: true
         |
         --- paidProducts
         |      |
         |      --- uid2
         |           |
         |           --- productId3: true
         |
         --- availableProducts
         |      |
         |      --- uid3
         |           |
         |           --- productId4: true

Using this database structure you can query your database for:

  1. All products (purchased, paid or available) that belong to a single user
  2. All users that have bought a particular product
  3. All purchased products that belong to a single user
  4. All paid products that belong to a single user
  5. All available products that belong to a single user

Every time a product is bought, you only need to move the product to the coresponding category. As AL. said, you could set your app to get the products by using the Firebase SDK. That would not be a problem.

This is called in Firebase denormalization is very normal with the Firebase Database. For this, i recomand see this video. If you want something more complex, i recomand you read this post, Structuring your Firebase Data correctly for a Complex App.

Using the Storage stuff from Firebase will store my products?

Yes, you can use Firebase Cloud Storage which is built for app developers who need to store and serve user-generated content, such as photos or videos.

Also I'll need to integrate the Authentication, right?

Yes, you need to use Firebase Authentication because you need to know the identity of a user. Knowing a user's identity allows an app to securely save user data in the cloud and provide the same personalized experience across all of the user's devices. Firebase Authentication provides backend services, easy-to-use SDKs, and ready-made UI libraries to authenticate users to your app. It supports authentication using passwords, phone numbers, popular federated identity providers like Google, Facebook and Twitter, and more.

The products can be dynamics? I mean, you can create for example an "event" and put that product only for x days?

Yes, you can do that using Cloud Functions For Firebase which lets you automatically run backend code in response to events triggered by Firebase features and HTTPS requests. Your code is stored in Google's cloud and runs in a managed environment. There's no need to manage and scale your own servers.

This is how you can query the database to display the product name and the price of all products.

DatabaseReference rootRef = FirebaseDatabase.getInstance().getReference();
DatabaseReference yourRef = rootRef.child("products");
ValueEventListener eventListener = new ValueEventListener() {
    @Override
    public void onDataChange(DataSnapshot dataSnapshot) {
        for(DataSnapshot ds : dataSnapshot.getChildren()) {
            String productName = ds.child("productName").getValue(String.class);
            double price = ds.child("price").getValue(Double.class);
            Log.d("TAG", productName + price);
        }
    }

    @Override
    public void onCancelled(DatabaseError databaseError) {}
};
yourRef.addListenerForSingleValueEvent(eventListener);

And this how you can query your database to see all purchasedProducts that belong to a specific user:

DatabaseReference rootRef = FirebaseDatabase.getInstance().getReference();
DatabaseReference uidRef = rootRef.child("purchasedProducts").child("uid1");
ValueEventListener eventListener = new ValueEventListener() {
    @Override
    public void onDataChange(DataSnapshot dataSnapshot) {
        for(DataSnapshot ds : dataSnapshot.getChildren()) {
            String productKey = ds.getKey();
            Log.d("TAG", productKey);
        }
    }

    @Override
    public void onCancelled(DatabaseError databaseError) {}
};
uidRef.addListenerForSingleValueEvent(eventListener);

Original Thread

By anonymous    2017-11-27

When using Firebase, there are a few rules that we need to keep in mind. In your particular case, i recomand you using denormalization and to change the database to be as flatten as possible. For a better understanding, i recomand you see this video, Denormalization is normal with the Firebase Database. Because Firebase Realtime Database cannot query over multiple properties, it ussaly involves duplication data. So this things are normal when it comes to Firebase. In your particular case, i recomand you doing both things.

I also recomand you read this post, Structuring your Firebase Data correctly for a Complex App. It's very well explained and will help you take the best decision.

You need to know that there is no perfect sturcture for a database. You need to structure your database in a way that allows you to read/write data very easily and in a very efficient way. If you don't know yet, Firebase has launched a new product named Cloud Firestore. Realtime Database does not scale automatically while the new Firestore does.

Original Thread

By anonymous    2017-12-04

You're almost there with your Firebase database structure, but because in Firebase an important rule is to have the data as flatten as possibile, i recomand you using denormalization. For this i recomand you see this video, Denormalization is normal with the Firebase Database. So in your case, you need to move user_connections node outside user node. Because Firebase can hold boolean values, the best practice is to use true and false instead of Y and N.

Firebase-root
    |
    --- userConnections
             |
             --- username1
                    |
                    --- username3: true
                    |
                    --- username4: true

In Firebase, when we attach a listener on a specific node, the entire node is downloaded. So if you want to display for example only the name of the users, you are downloading user connections too, which is not neccessay in this case. So this is the main reason to have a database structure which looks like this.

If you want to get data from all users that belong to a particular user, you need to query your database twice and this how can be achieved this:

DatabaseReference rootRef = FirebaseDatabase.getInstance().getReference();
DatabaseReference userNameRef = rootRef.child("users").child("userName1");
ValueEventListener valueEventListener = new ValueEventListener() {
    @Override
    public void onDataChange(DataSnapshot dataSnapshot) {
        for(DataSnapshot ds : dataSnapshot.getChildren()) {
            String userName = ds.getKey();

            DatabaseReference usersRef = rootRef.child("users").child(userName);
            ValueEventListener eventListener = new ValueEventListener() {
                @Override
                public void onDataChange(DataSnapshot dataSnapshot) {
                    String email_address = ds.child("email_address").getValue(String.class);
                    Log.d("TAG", email_address);
                }

                @Override
                public void onCancelled(DatabaseError databaseError) {}
            };
            usersRef.addListenerForSingleValueEvent(eventListener);
        }
    }

    @Override
    public void onCancelled(DatabaseError databaseError) {}
};
userNameRef.addListenerForSingleValueEvent(valueEventListener);

With this code, first you are getting all user names that belong to that particular user named userName1. Second we query only those users to display their email addresses.

Original Thread

By anonymous    2017-12-04

The second solution is better because you'll save bandwith. This practice is called in denormalization and is a common practice when it comes to Firebase. The first solution is not good becase every time you want to display the users you donwload unnecessary data. If you want to read more details about how you can structure a Firebase database in a efficient way, please read this post, Structuring your Firebase Data correctly for a Complex App. Also, you can take a look a this tutorial, Denormalization is normal with the Firebase Database, for a better understanding.

Original Thread

By anonymous    2017-12-11

I'm currently using Ionic CLI 3.19 with Cordova CLI 7.1.0 (@ionic-app-script 3.1.4)

The problem that I’m currently facing with is, I should update friends node values simultaneously every time the related data get changed from elsewhere. I’d like to clarify my objective with some screenshots to make it more clear.

As you can see from the image below, each child node consists of a user array that has a user id as a key of friends node. The reason why I store as an array is because each user could have many friends. In this example, Jeff Kim has one friend which is John Doe vice versa.

friends node image

Every time when one of the users data get changed for some reason, I want the related data in friends node also want them to be updated too.

For example, when Jeff Kim changed his profile photo or statusMessage all the same uid that reside in friends node which matches with Jeff Kim’s uid need to be updated based on what user has changed.

users node image

user-service.ts

    constructor(private afAuth: AngularFireAuth, private afDB: AngularFireDatabase,){
      this.afAuth.authState.do(user => {
      this.authState = user;
        if (user) {
          this.updateOnConnect();
          this.updateOnDisconnect();
        }
      }).subscribe();
     }
    updateOnConnect() {
      return this.afDB.object('.info/connected').valueChanges()
             .do(connected => {
             let status = connected ? 'online' : 'offline'
             this.updateCurrentUserActiveStatusTo(status)
             }).subscribe()
            }


    updateOnDisconnect() {
      firebase.database().ref().child(`users/${this.currentUserId}`)
              .onDisconnect()
              .update({currentActiveStatus: 'offline'});
            }

    sendFriendRequest(recipient: string, sender: User) {
      let senderInfo = {
      uid: sender.uid,
      displayName: sender.displayName,
      photoURL: sender.photoURL,
      statusMessage: sender.statusMessage,
      currentActiveStatus: sender.currentActiveStatus,
      username: sender.username,
      email: sender.email,
      timestamp: Date.now(),
      message: 'wants to be friend with you.'
    }
    return new Promise((resolve, reject) => {
      this.afDB.list(`friend-requests/${recipient}`).push(senderInfo).then(() => {
      resolve({'status': true, 'message': 'Friend request has sent.'});
     }, error => reject({'status': false, 'message': error}));
  });
}

    fetchFriendRequest() {
    return this.afDB.list(`friend-requests/${this.currentUserId}`).valueChanges();
  }

    acceptFriendRequest(sender: User, user: User) {
      let acceptedUserInfo = {
      uid: sender.uid,
      displayName: sender.displayName,
      photoURL: sender.photoURL,
      statusMessage: sender.statusMessage,
      currentActiveStatus: sender.currentActiveStatus,
      username: sender.username,
      email: sender.email
     }
     this.afDB.list(`friends/${sender.uid}`).push(user); 
     this.afDB.list(`friends/${this.currentUserId}`).push(acceptedUserI
     this.removeCompletedFriendRequest(sender.uid);
}

According to this clip that I've just watched, it looks like I did something called Denormalization and the solution might be using Multi-path updates to change data with consistency. Data consistency with Multi-path updates. However, it's kinda tricky to fully understand and start writing some code.

I've done some sort of practice to make sure update data in multiple locations without calling .update method twice.

// I have changed updateUsername method from the code A to code B
// Code A
updateUsername(username: string) {
  let data = {};
  data[username] = this.currentUserId;
  this.afDB.object(`users/${this.currentUserId}`).update({'username': username});
  this.afDB.object(`usernames`).update(data);
}
// Code B
updateUsername(username: string) {
  const ref = firebase.database().ref(); 
  let updateUsername = {};
  updateUsername[`usernames/${username}`] = this.currentUserId; 
  updateUsername[`users/${this.currentUserId}/username`] = username;
  ref.update(updateUsername);
}

Original Thread

By anonymous    2017-12-11

First of all, when we are referring to Firebase, we cannot speak about tables. In a NoSQL database there are no tables. There are only pairs of key and value.

To solve your problem, i recomand you change your database a little bit, by adding a new node in your Firebase database named userAdmins. When you add a user into your database and is also an admin, add him in this new section. Your new node should look like this:

Firebase-root
    |
    --- userAdmins
            |
            --- userId1: true
            |
            --- userId2: true

This practice is named denormalization and for that i recomand you see this video, Denormalization is normal with the Firebase Database. If you'll see this video, you'll have a better understanding about this practice.

To verify if a user is admin, just use exists() method on the DataSnapshot object like this:

DatabaseReference rootRef = FirebaseDatabase.getInstance().getReference();
DatabaseReference uidRef = rootRef.child("userAdmins").child(uid);
ValueEventListener eventListener = new ValueEventListener() {
    @Override
    public void onDataChange(DataSnapshot dataSnapshot) {
        if(dataSnapshot.exists()) {
            //do something
        } else {
            //do something else
        }
    }

    @Override
    public void onCancelled(DatabaseError databaseError) {}
};
uidRef.addListenerForSingleValueEvent(eventListener);

In which uid is the id of the user you want to verify if is an admin.

Original Thread

By anonymous    2017-12-18

To achieve this, you don't need to restructure your database entirely, you just need to change it a little bit. To solve this problem, you need to add another node to your database named usClients. Every time you add a new user which is from USA, add it also in this new created node. Your new node should look like this:

Firebase-root
    |
    --- usClients
           |
           --- usClientId1 : true
           |
           --- usClientId2 : true

Whith this structure you can query your database to get only the clients within USA. This can be done attaching a listener on usClients node and iteration on the DataSnapshot object.

This practice is called denormalization and is a common practice when it comes to Firebase. For a better understanding, i recomand you see this video, Denormalization is normal with the Firebase Database.

Original Thread

By anonymous    2017-12-18

I think you are right doing this denormalization, and your multi-path updates is in the right direction. But assuming several users can have several friends, I miss a loop in friends' table.

You should have tables users, friends and a userFriend. The last table is like a shortcut to find user inside friends, whitout it you need to iterate every friend to find which the user that needs to be updated.

I did a different approach in my first_app_example [angular 4 + firebase]. I removed the process from client and added it into server via onUpdate() in Cloud functions.

In the code bellow when user changes his name cloud function executes and update name in every review that the user already wrote. In my case client-side does not know about denormalization.

//Executed when user.name changes
exports.changeUserNameEvent = functions.database.ref('/users/{userID}/name').onUpdate(event =>{
    let eventSnapshot = event.data;
    let userID = event.params.userID;
    let newValue = eventSnapshot.val();

    let previousValue = eventSnapshot.previous.exists() ? eventSnapshot.previous.val() : '';

    console.log(`[changeUserNameEvent] ${userID} |from: ${previousValue} to: ${newValue}`);

    let userReviews = eventSnapshot.ref.root.child(`/users/${userID}/reviews/`);
    let updateTask = userReviews.once('value', snap => {
    let reviewIDs = Object.keys(snap.val());

    let updates = {};
    reviewIDs.forEach(key => { // <---- note that I loop in review. You should loop in your userFriend table
        updates[`/reviews/${key}/ownerName`] = newValue;
    });

    return eventSnapshot.ref.root.update(updates);
    });

    return updateTask;
});

EDIT

Q: I structured friends node correctly or not

I prefer to replicate (denormalize) only the information that I need more often. Following this idea, you should just replicate 'userName' and 'photoURL' for example. You can aways access all friends' information in two steps:

 let friends: string[];
 for each friend in usrService.getFriend(userID)
    friends.push(usrService.getUser(friend))

Q: you mean I should create a Lookup table?

The clip mentioned in your question, David East gave us an example how to denormalize. Originaly he has users and events. And in denormalization he creates eventAttendees that is like a vlookup (like you sad).

Q: Could you please give me an example?

Sure. I removed some user's information and add an extra field friendshipTypes

users
    xxsxaxacdadID1
        currentActiveStatus: online
        email: zinzzkak@gmail.com
        gender: Male
        displayName: Jeff Kim
        photoURL: https://firebase....
        ...
    trteretteteeID2
        currentActiveStatus: online
        email: hahehahaheha@gmail.com
        gender: Male
        displayName: Joeh Doe
        photoURL: https://firebase....
        ...

friends
    xxsxaxacdadID1
        trteretteteeID2
            friendshipTypes: bestFriend //<--- extra information
            displayName: Jeff Kim
            photoURL: https://firebase....
    trteretteteeID2
        xxsxaxacdadID1
            friendshipTypes: justAfriend //<--- extra information
            displayName: John Doe
            photoURL: https://firebase....


userfriends
    xxsxaxacdadID1
        trteretteteeID2: true
        hgjkhgkhgjhgID3: true
    trteretteteeID2
        trteretteteeID2: true

Original Thread

Recommended Books

    Submit Your Video

    If you have some great dev videos to share, please fill out this form.