@sun are you trying to use an autoencoder or something for upscaling? I could try and link some study(ies) on existing networks that work and have existing data on their efficiency instead of hitting your head against the table like that. It would also make it easier to implement them.